Extended reality for productivity

ABSTRACT

An integrated computational interface device may include a portable housing having a key region and a non-key region; a keyboard associated with the key region of the housing; and a holder associated with the non-key region of the housing. The holder may be configured for selective engagement with and disengagement from the wearable extended reality appliance, such that when the wearable extended reality appliance is selectively engaged with the housing via the holder, the wearable extended reality appliance is transportable with the housing.

CROSS REFERENCES TO RELATED APPLICATIONS

This application claims the benefit of priority of U.S. ProvisionalPatent Application No. 63/147,051, filed on Feb. 8, 2021, U.S.Provisional Patent Application No. 63/157,768, filed on Mar. 7, 2021,U.S. Provisional Patent Application No. 63/173,095, filed on Apr. 9,2021, U.S. Provisional Patent Application No. 63/213,019, filed on filedon Jun. 21, 2021, U.S. Provisional Patent Application No. 63/215,500,filed on Jun. 27, 2021, U.S. Provisional Patent Application No.63/216,335, filed on Jun. 29, 2021, U.S. Provisional Patent ApplicationNo. 63/226,977, filed on Jul. 29, 2021, U.S. Provisional PatentApplication No. 63/300,005, filed on Jan. 16, 2022, U.S. ProvisionalPatent Application No. 63/307,207, filed on Feb. 7, 2022, U.S.Provisional Patent Application No. 63/307,203, filed on Feb. 7, 2022,and U.S. Provisional Patent Application No. 63/307,217, filed on Feb. 7,2022, all of which are incorporated herein by reference in theirentirety.

BACKGROUND I. Technical Field

The present disclosure generally relates to the field of extendedreality. More specifically, the present disclosure relates to systems,methods, and devices for providing productivity applications using anextended reality environment.

II. BACKGROUND INFORMATION

For many years, PC users were faced with a productivity dilemma: eitherto limit their mobility (when selecting a desktop computer) or to limittheir screen size (when selecting a laptop computer). One partialsolution to this dilemma is using a docking station. A docking stationis an interface device for connecting a laptop computer with otherdevices. By plugging the laptop computer into the docking station,laptop users can enjoy the increased visibility provided by a largermonitor. But because the large monitor is stationary, the mobility ofthe user—while improved—is still limited. For example, even laptop userswith docking stations do not have the freedom of using two 32″ screensanywhere they want.

Some of the disclosed embodiments are directed to providing a newapproach for solving the productivity dilemma, one that uses extendedreality (XR) to provide a mobile environment that enables users toexperience the comfort of a stationary workspace anywhere they want byproviding virtual desktop-like screens.

SUMMARY

Embodiments consistent with the present disclosure provide systems,methods, and devices for providing and supporting productivityapplications using an extended reality environment.

Some disclosed embodiments may include an integrated computationalinterface device may include a portable housing having a key region anda non-key region; a keyboard associated with the key region of thehousing; and a holder associated with the non-key region of the housing.The holder may be configured for selective engagement with anddisengagement from a wearable extended reality appliance, such that whenthe wearable extended reality appliance is selectively engaged with thehousing via the holder, the wearable extended reality appliance istransportable with the housing.

Some disclosed embodiments may include an integrated computationalinterface device includes a housing, at least one image sensor, and afoldable protective cover. The housing may have a key region and anon-key region, and a keyboard associated with the key region. Thefoldable protective cover incorporates the at least one image sensor.The protective cover may be configured to be manipulated into aplurality of folding configurations, including a first foldingconfiguration, wherein the protective cover may be configured to encasethe key region and at least a portion of the non-key region, and asecond folding configuration, wherein the protective cover may beconfigured to stand in a manner that causes an optical axis of the atleast one image sensor to generally face a user of the integratedcomputational interface device while the user types on the keyboard.

Some disclosed embodiments may include a case for an integratedcomputational interface device includes at least one image sensor and afoldable protective cover incorporating the at least one image sensor.The protective cover may be configured to be manipulated into aplurality of folding configurations. In a first folding configuration,the protective cover may be configured to encase a housing of theintegrated computational interface device having a key region and anon-key region. In a second folding configuration, the protective covermay be configured to stand in a manner that causes an optical axis ofthe at least one image sensor to generally face a user of the integratedcomputational interface device while the user types on a keyboardassociated with the key region.

Some disclosed embodiments may include systems, methods andnon-transitory computer readable media for changing display of virtualcontent based on temperature. Some of these embodiments may involvedisplaying virtual content via a wearable extended reality appliance,wherein during displaying of the virtual content, heat is generated byat least one component of the wearable extended reality appliance;receiving information indicative of a temperature associated with thewearable extended reality appliance; determining a need to changedisplay settings of the virtual content based on the receivedinformation; and based on the determination, changing the displaysettings of the virtual content to achieve a target temperature.

Some disclosed embodiments may include systems, methods andnon-transitory computer readable media for implementing hybrid virtualkeys in an extended reality environment. Some of these embodiments mayinvolve receiving, during a first time period, first signalscorresponding to positions on a touch-sensitive surface of a pluralityof virtual activatable elements virtually projected by a wearableextended reality appliance on the touch-sensitive surface; determiningfrom the first signals the positions of the plurality of virtualactivatable elements on the touch-sensitive surface; receiving a touchinput of a user via the touch-sensitive surface, wherein the touch inputincludes second signals generated as a result of interactions with atleast one sensor within the touch-sensitive surface; determining acoordinate location associated with the touch input based on the secondsignals generated as the result of the interactions with the at leastone sensor within the touch-sensitive surface; comparing the coordinatelocation of the touch input with at least one of the determinedpositions to identify one of the plurality of virtual activatableelements corresponding to the touch input; and causing a change in avirtual content associated with the wearable extended reality appliance,wherein the change corresponds to the identified one of the plurality ofvirtual activatable elements.

Some disclosed embodiments may include systems, methods andnon-transitory computer readable media for controlling a virtual displayusing a keyboard and a wearable extended reality appliance combination.Some of these embodiments may involve receiving, from a firsthand-position sensor associated with the wearable extended realityappliance, first signals representing first hand-movements; receiving,from a second hand-position sensor associated with the keyboard, secondsignals representing second hand-movements, wherein the secondhand-movements include actions other than interactions with a feedbackcomponent; and controlling the virtual display based on the firstsignals and the second signals.

Some disclosed embodiments may include systems, methods andnon-transitory computer readable media for integrating a moveable inputdevice with a virtual display projected via a wearable extended realityappliance. Some of these embodiments may involve receiving motionsignals associated with the moveable input device, the motion signalsreflecting physical movement of the moveable input device; outputtingduring a first time period, first display signals to the wearableextended reality appliance, the first display signals being configuredto cause the wearable extended reality appliance to virtually presentcontent in a first orientation; outputting during a second time perioddifferent from the first time period, second display signals to thewearable extended reality appliance, the second display signals beingconfigured to cause the wearable extended reality appliance to virtuallypresent the content in a second orientation different from the firstorientation; and switching between the output of the first displaysignals and the output of the second display signals based on thereceived motions signals of the moveable input device.

Some disclosed embodiments may include systems, methods andnon-transitory computer readable media for virtually extending aphysical keyboard. Some of these embodiments may involve receiving imagedata from an image sensor associated with a wearable extended realityappliance, the image data representing a keyboard placed on a surface;determining that the keyboard is paired with the wearable extendedreality appliance; receiving an input for causing a display of a virtualcontroller in conjunction with the keyboard; displaying, via thewearable extended reality appliance, the virtual controller in a firstlocation on the surface, wherein in the first location, the virtualcontroller has an original spatial orientation relative to the keyboard;detecting a movement of the keyboard to a different location on thesurface; and in response to the detected movement of the keyboard,presenting the virtual controller in a second location on the surface,wherein in the second location, a subsequent spatial orientation of thevirtual controller relative to the keyboard corresponds to the originalspatial orientation.

Some disclosed embodiments may include systems, methods andnon-transitory computer readable media for coordinating virtual contentdisplay with mobility status. Some of these embodiments may involveaccessing rules associating a plurality of user mobility statuses with aplurality of display modes for presenting virtual content via a wearableextended reality appliance; receiving first sensor data from at leastone sensor associated with the wearable extended reality appliance, thefirst sensor data being reflective of a mobility status of a user of thewearable extended reality appliance during a first time period; based onthe first sensor data, determining that during the first time period theuser of the wearable extended reality appliance is associated with afirst mobility status; implementing at least a first accessed rule togenerate a first display of the virtual content via the wearableextended reality appliance associated with the first mobility status;receiving second sensor data from the at least one sensor, the secondsensor data being reflective of the mobility status of the user during asecond time period; based on the second sensor data, determining thatduring the second time period the user of the wearable extended realityappliance is associated with a second mobility status; and implementingat least a second accessed rule to generate a second display of thevirtual content via the wearable extended reality appliance associatedwith the second mobility status, wherein the second display of thevirtual content differs from the first display of the virtual content.

Some disclosed embodiments may include systems, methods andnon-transitory computer readable media for modifying display of virtualobjects docked to a movable input device. Some of these embodiments mayinvolve receiving image data from an image sensor associated with awearable extended reality appliance, the image data representing aninput device placed at a first location on a supporting surface; causingthe wearable extended reality appliance to generate a presentation of atleast one virtual object in proximity to the first location; docking theat least one virtual object to the input device; determining that theinput device is in a second location on the support surface; in responseto the determination that the input device is in the second location,updating the presentation of the at least one virtual object such thatthe at least one virtual object appears in proximity to the secondlocation; determining that the input device is in a third locationremoved from the support surface; and in response to the determinationthat the input device is removed from the support surface, modifying thepresentation of the at least one virtual object.

Some disclosed embodiments may include systems, methods andnon-transitory computer readable media for docking virtual objects tovirtual display screens in an extended reality environment. Some ofthese embodiments may involve generating virtual content forpresentation via a wearable extended reality appliance, where thevirtual content includes a virtual display and a plurality of virtualobjects located outside the virtual display; receiving a selection of atleast one virtual object from the plurality of virtual objects; dockingthe at least one virtual object to the virtual display; after dockingthe at least one virtual object to the virtual display, receiving aninput indicative of an intent to change a location of the virtualdisplay without an expression of an intent to move the at least onevirtual object; changing the location of the virtual display in responseto the input; and wherein changing the location of the virtual display,causes the at least one virtual object to move with the virtual displayas a result of the docking of the at least one virtual object to thevirtual display.

Some disclosed embodiments may include systems, methods andnon-transitory computer readable media for implementing selectivevirtual object display changes. Some of these embodiments may involvegenerating an extended reality environment via a wearable extendedreality appliance, the extended reality environment including a firstvirtual plane associated with a physical object and a second virtualplane associated with an item, the second virtual plane extending in adirection transverse to the first virtual plane; accessing a firstinstruction for docking a first set of virtual objects in firstpositions associated with the first virtual plane; accessing a secondinstruction for docking a second set of virtual objects in secondpositions associated the second virtual plane; receiving a first inputassociated with a movement of the physical object; in response toreceiving the first input, causing a change in a display of the firstset of virtual objects in a manner corresponding the movement of thephysical object while maintaining the second set of virtual objects inthe second positions; receiving a second input associated with amovement of the item; and in response to receiving the second input,causing a change in a display of the second set of virtual objects in amanner corresponding the movement of the item while maintaining thefirst positions of the first set of virtual objects.

Some disclosed embodiments may include systems, methods andnon-transitory computer readable media for determining a displayconfiguration for presenting virtual content. Some of these embodimentsmay involve receiving image data from an image sensor associated with awearable extended reality appliance, wherein the wearable extendedreality appliance is configured to be paired with multiple input devicesand each input device is associated with default display settings;analyzing the image data to detect a particular input device placed on asurface; determining a value of at least one use parameter for theparticular input device; retrieving from memory default display settingsassociated with the particular input device; determining a displayconfiguration for presenting virtual content based on the value of theat least one use parameter and the retrieved default display settings;and causing a presentation of the virtual content via the wearableextended reality appliance according to the determined displayconfiguration.

Some disclosed embodiments may include systems, methods andnon-transitory computer readable media for augmenting a physical displaywith a virtual display. Some of these embodiments may involve receivingfirst signals representative of a first object fully presented on aphysical display; receiving second signals representative of a secondobject having a first portion presented on the physical display and asecond portion extending beyond a border of the physical display;receiving third signals representative of a third object initiallypresented on the physical display and subsequently wholly moved beyondthe border of the physical display; in response to receipt of the secondsignals, causing the second portion of the second object to be presentedvia a wearable extended reality appliance in a virtual space while thefirst portion of the second object is presented on the physical display;and in response to receipt of the third signals, causing the thirdobject to be wholly presented via the wearable extended realityappliance in the virtual space following the third object having beenwholly presented on the physical display.

Consistent with other disclosed embodiments, non-transitorycomputer-readable storage media may store program instructions, whichare executed by at least one processing device and perform any of themethods described herein.

The foregoing general description and the following detailed descriptionare exemplary and explanatory only and are not restrictive of theclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this disclosure, illustrate various disclosed embodiments. Inthe drawings:

FIG. 1 is a schematic illustration of a user, using an example extendedreality system, consistent with some embodiments of the presentdisclosure.

FIG. 2 is a schematic illustration of the main components of the exampleextended reality system of FIG. 1, consistent with some embodiments ofthe present disclosure.

FIG. 3 is a block diagram illustrating some of the components of aninput unit, consistent with some embodiments of the present disclosure.

FIG. 4 is a block diagram illustrating some of the components of anextended reality unit, consistent with some embodiments of the presentdisclosure.

FIG. 5 is a block diagram illustrating some of the components of aremote processing unit, consistent with some embodiments of the presentdisclosure.

FIG. 6 is a top view of an exemplary first embodiment of an integratedcomputational interface device with a wearable extended realityappliance selectively engaged with the integrated computationalinterface device.

FIG. 7A is a top view of an exemplary second embodiment of an integratedcomputational interface device with a wearable extended realityappliance selectively engaged with the integrated computationalinterface device.

FIG. 7B is a left side view of the second exemplary embodiment of theintegrated computational interface device shown in FIG. 7A.

FIG. 8A is a front perspective view of a wearable extended realityappliance selectively engaged with a first exemplary embodiment of aholder.

FIG. 8B is a rear perspective view of the wearable extended realityappliance selectively disengaged from the first exemplary embodiment ofthe holder shown in FIG. 8A.

FIG. 9A is a front perspective view of a wearable extended realityappliance selectively engaged with a second exemplary embodiment of aholder.

FIG. 9B is a rear perspective view of the wearable extended realityappliance selectively disengaged from the second exemplary embodiment ofthe holder shown in FIG. 9A.

FIG. 10A is a front perspective view of a wearable extended realityappliance selectively engaged with a third exemplary embodiment of aholder.

FIG. 10B is a rear perspective view of the wearable extended realityappliance selectively disengaged from the third exemplary embodiment ofthe holder shown in FIG. 10A.

FIG. 11A is a front perspective view of a wearable extended realityappliance selectively engaged with a fourth exemplary embodiment of aholder.

FIG. 11B is a rear perspective view of the wearable extended realityappliance selectively disengaged from the fourth exemplary embodiment ofthe holder shown in FIG. 11A.

FIG. 12A is a top view of a third exemplary embodiment of an integratedcomputational interface device with a wearable extended realityappliance selectively engaged with the integrated computationalinterface device.

FIG. 12B is a left side view of the third exemplary embodiment of theintegrated computational interface device shown in FIG. 12A.

FIG. 13A is a side perspective view of an exemplary integratedcomputational interface device having a protective cover in a firstencasing mode, consistent with some embodiments of the presentdisclosure.

FIG. 13B is a left side perspective view of the integrated computationalinterface device of FIG. 13A having a protective cover in a secondencasing mode, consistent with some embodiments of the presentdisclosure.

FIG. 14 is a side perspective view of a second exemplary embodiment ofan integrated computational interface device having a protective coverin a second encasing mode.

FIG. 15 is a front perspective view of a first exemplary embodiment ofan integrated computational interface device.

FIG. 16 is a front perspective view of a second exemplary embodiment ofan integrated computational interface device.

FIG. 17 is a top view of a third exemplary embodiment of an integratedcomputational interface device.

FIG. 18 is a front perspective view of a fourth exemplary embodiment ofan integrated computational interface device with a foldable protectivecover in a first folding configuration.

FIG. 19 is an exploded view of a portion of an exemplary embodiment of afoldable protective cover.

FIG. 20 is a side view of a fifth exemplary embodiment of an integratedcomputational interface device.

FIG. 21 is a side view of a sixth exemplary embodiment of an integratedcomputational interface device.

FIG. 22 is a front perspective view of a seventh exemplary embodiment ofan integrated computational interface device.

FIG. 23 is a block diagram illustrating exemplary working parametersportions of a wearable extended reality appliance, consistent with somedisclosed embodiments.

FIG. 24 is an exemplary chart illustrating variation of display settingsbased on the heat-emitting light source temperature over time,consistent with some disclosed embodiments.

FIG. 25 illustrates an example of reducing a display size of a portionof the virtual content based on received temperature information,consistent with some disclosed embodiments.

FIG. 26 is a flow chart illustrating an exemplary method for changingdisplay settings based on a temperature of a wearable extended realityappliance, consistent with some embodiments of the present disclosure.

FIG. 27 illustrates an example of a wearable extended reality appliancevirtually projecting onto a touch-sensitive surface, consistent withsome embodiments of the present disclosure.

FIG. 28 illustrates an example of a keyboard and a touch-sensitivesurface, consistent with some embodiments of the present disclosure.

FIG. 29 illustrates an example of a user interacting with atouch-sensitive surface, consistent with some embodiments of the presentdisclosure.

FIG. 30 illustrates an example of a user interacting with atouch-sensitive surface to navigate a cursor, consistent with someembodiments of the present disclosure.

FIG. 31 illustrates a flow chart of an exemplary method for implementinghybrid virtual keys in an extended reality environment, consistent withsome embodiments of the present disclosure.

FIG. 32 illustrates an example of a keyboard with additional virtualactivatable elements virtually projected onto the keys of the keyboard,consistent with some embodiments of the present disclosure.

FIG. 33 illustrates an example of a keyboard and a wearable extendedreality appliance combination to control a virtual display, consistentwith some embodiments of the present disclosure.

FIG. 34 illustrates an example of a first hand-position sensorassociated with a wearable extended reality appliance, consistent withsome embodiments of the present disclosure.

FIG. 35 illustrates an example of a second hand-position sensorassociated with a keyboard, consistent with some embodiments of thepresent disclosure.

FIG. 36 illustrates an example of different types of first and secondhand-position sensors, consistent with some embodiments of the presentdisclosure.

FIG. 37 illustrates an example of a keyboard that includes an associatedinput area including a touch pad and keys, consistent with someembodiments of the present disclosure.

FIG. 38 illustrates an example of a wearable extended reality applianceselectively connectable to a keyboard via a connector, consistent withsome embodiments of the present disclosure.

FIG. 39 illustrates an exemplary virtual display with a moveable inputdevice at a first time period, consistent with some embodiments of thepresent disclosure.

FIG. 40 illustrates an exemplary virtual display with a moveable inputdevice at a second time period, consistent with some embodiments of thepresent disclosure.

FIG. 41 illustrates an exemplary virtual display and a type of movementof a moveable input device, consistent with some embodiments of thepresent disclosure.

FIG. 42 illustrates an exemplary virtual display in a first orientationrelative to a moveable input device before a first time period,consistent with some embodiments of the present disclosure.

FIG. 43 illustrates a change in the size of an exemplary virtual displaybased on motion signals associated with a moveable input device,consistent with some embodiments of the present disclosure.

FIG. 44 illustrates an exemplary virtual display configured to enablevisual presentation of textual input entered using a moveable inputdevice, consistent with some embodiments of the present disclosure.

FIG. 45A illustrates an exemplary process for integrating a moveableinput device with a virtual display projected via a wearable extendedreality appliance, consistent with some embodiments of the presentdisclosure.

FIG. 45B illustrates another exemplary process for integrating amoveable input device with a virtual display projected via a wearableextended reality appliance, consistent with some embodiments of thepresent disclosure.

FIG. 46 illustrates an example of a keyboard and a virtual controller,consistent with some embodiments of the present disclosure.

FIG. 47 illustrates an example of a keyboard and a virtual controllermoved from one location to another location, consistent with someembodiments of the present disclosure.

FIG. 48 illustrates another example of a keyboard and a virtualcontroller moved from one location to another location, consistent withsome embodiments of the present disclosure.

FIG. 49 is a block diagram of an exemplary process for virtuallyextending a physical keyboard, consistent with some embodiments of thepresent disclosure.

FIGS. 50A to 50D illustrate examples of various virtual content displayscoordinated with different mobility statuses, consistent with someembodiments of the present disclosure.

FIGS. 51A and 51B illustrate examples of different display modesassociated with different types of virtual objects for differingmobility statuses, consistent with some embodiments of the presentdisclosure.

FIGS. 52A and 52B illustrate examples of different display modesassociated with differing mobility statuses based on an environmentalcontext, consistent with some embodiments of the present disclosure.

FIG. 53 is a flowchart of an exemplary method for coordinating virtualcontent display with mobility status, consistent with some embodimentsof the present disclosure.

FIG. 54 generally illustrates a docking concept, consistent with somedisclosed embodiments.

FIG. 55A is an exemplary illustration of a keyboard docked with avirtual object at a first location on a supporting surface, consistentwith some disclosed embodiments.

FIG. 55B is an exemplary illustration of a keyboard docked with avirtual object at a second location on a supporting surface, consistentwith some disclosed embodiments.

FIG. 56A is an exemplary illustration of a keyboard being moved from alocation on a supporting surface to a location that is not on asupporting surface, wherein one or more presented virtual objects aremodified, consistent with some disclosed embodiments.

FIG. 56B is an exemplary illustration of a keyboard being moved from alocation on a supporting surface to a location that is not on asupporting surface, wherein one or more presented virtual objectsdisappear, consistent with some disclosed embodiments.

FIG. 57 is a flow chart illustrating an exemplary method for evolvingdocking based on detected keyboard positions, consistent with somedisclosed embodiments.

FIG. 58 illustrates an example of a virtual display and a docked virtualobject representing a phone of a user, consistent with some embodimentsof the present disclosure.

FIGS. 59A and 59B illustrate examples of a virtual display and aplurality of virtual objects located outside the virtual display beforeand after the virtual display changes locations, consistent with someembodiments of the present disclosure.

FIGS. 60A and 60B, illustrate examples of a virtual display and aplurality of virtual objects docked to the virtual display and to othervirtual objects before and after the virtual display changes locations,consistent with some embodiments of the present disclosure.

FIGS. 61A and 61B, illustrate examples of a virtual display and aphysical object, consistent with some embodiments of the presentdisclosure.

FIGS. 62A and 62B, illustrate examples of a virtual display and aplurality of virtual objects before and after the virtual displaychanges locations, consistent with some embodiments of the presentdisclosure.

FIG. 63 illustrates a flow chart of an exemplary method for dockingvirtual objects to virtual display screens, consistent with someembodiments of the present disclosure.

FIG. 64 illustrates an example of a physical object in a first plane andan item in a second plane, consistent with some embodiments of thepresent disclosure.

FIG. 65 illustrates an example of virtual objects docked to positions ina virtual plane prior to movement of the physical object, consistentwith some embodiments of the present disclosure.

FIG. 66 illustrates an example of a movement of a physical object andvirtual objects, consistent with some embodiments of the presentdisclosure.

FIG. 67 illustrates an example of a movement of an item and virtualobjects, consistent with some embodiments of the present disclosure.

FIG. 68 illustrates a flow chart of an exemplary method that may beexecuted by a processor to perform operations for implementing selectivevirtual object display changes, consistent with some embodiments of thepresent disclosure.

FIG. 69 shows a schematic illustrating an exemplary wearable extendedreality appliance system, consistent with some embodiments of thepresent disclosure.

FIG. 70 shows a schematic illustrating an exemplary displayconfiguration, consistent with some embodiments of the presentdisclosure.

FIG. 71 shows a schematic illustrating another exemplary displayconfiguration, consistent with some embodiments of the presentdisclosure.

FIG. 72 shows a flowchart illustrating an exemplary process fordetermining a display configuration for presenting virtual content,consistent with some embodiments of the present disclosure.

FIG. 73 illustrates examples of virtual content displayed both in an outof a computer screen, consistent with some embodiments of the presentdisclosure.

FIG. 74 illustrates examples of virtual content displayed both in an outof a smart watch, consistent with some embodiments of the presentdisclosure.

FIG. 75A-D illustrate examples of the movement of virtual contentbetween computer screens, consistent with some embodiments of thepresent disclosure.

FIG. 76 is a flowchart illustrating an exemplary process for extending aworking display, consistent with some embodiments of the presentdisclosure.

DETAILED DESCRIPTION

The following detailed description refers to the accompanying drawings.Wherever possible, the same reference numbers are used in the drawingsand the following description to refer to the same or similar parts.While several illustrative embodiments are described herein,modifications, adaptations and other implementations are possible. Forexample, substitutions, additions, or modifications may be made to thecomponents illustrated in the drawings, and the illustrative methodsdescribed herein may be modified by substituting, reordering, removing,or adding steps to the disclosed methods. Accordingly, the followingdetailed description is not limited to the specific embodiments andexamples, but is inclusive of general principles described herein andillustrated in the figures in addition to the general principlesencompassed by the appended claims.

The present disclosure is directed to systems and methods for providingusers an extended reality environment. The term “extended realityenvironment,” which may also be referred to as “extended reality,”“extended reality space,” or “extended environment,” refers to all typesof real- and-virtual combined environments and human-machineinteractions at least partially generated by computer technology. Theextended reality environment may be a completely simulated virtualenvironment or a combined real- and-virtual environment that a user mayperceive from different perspectives. In some examples, the user mayinteract with elements of the extended reality environment. Onenon-limiting example of an extended reality environment may be a virtualreality environment, also known as “virtual reality” or a “virtualenvironment.” An immersive virtual reality environment may be asimulated non-physical environment which provides to the user theperception of being present in the virtual environment. Anothernon-limiting example of an extended reality environment may be anaugmented reality environment, also known as “augmented reality” or“augmented environment.” An augmented reality environment may involvelive direct or indirect view of a physical real-world environment thatis enhanced with virtual computer-generated perceptual information, suchas virtual objects that the user may interact with. Another non-limitingexample of an extended reality environment is a mixed realityenvironment, also known as “mixed reality” or a “mixed environment.” Amixed reality environment may be a hybrid of physical real-world andvirtual environments, in which physical and virtual objects may coexistand interact in real time. In some examples, both augmented realityenvironments and mixed reality environments may include a combination ofreal and virtual worlds, real-time interactions, and accurate 3Dregistration of virtual and real objects. In some examples, bothaugmented reality environment and mixed reality environments may includeconstructive overlaid sensory information that may be added to thephysical environment. In other examples, both augmented realityenvironment and mixed reality environments may include destructivevirtual content that may mask at least part of the physical environment.

In some embodiments, the systems and methods may provide the extendedreality environment using an extended reality appliance. The termextended reality appliance may include any type of device or system thatenables a user to perceive and/or interact with an extended realityenvironment. The extended reality appliance may enable the user toperceive and/or interact with an extended reality environment throughone or more sensory modalities. Some non-limiting examples of suchsensory modalities may include visual, auditory, haptic, somatosensory,and olfactory. One example of the extended reality appliance is avirtual reality appliance that enables the user to perceive and/orinteract with a virtual reality environment. Another example of theextended reality appliance is an augmented reality appliance thatenables the user to perceive and/or interact with an augmented realityenvironment. Yet another example of the extended reality appliance is amixed reality appliance that enables the user to perceive and/orinteract with a mixed reality environment.

Consistent with one aspect of the disclosure, the extended realityappliance may be a wearable device, such as a head-mounted device, forexample, smart glasses, smart contact lens, headsets or any other deviceworn by a human for purposes of presenting an extended reality to thehuman. Other extended reality appliances may include holographicprojector or any other device or system capable of providing anaugmented reality (AR), virtual reality (VR), mixed reality (MR), or anyimmersive experience. Typical components of wearable extended realityappliances may include at least one of: a stereoscopic head-mounteddisplay, a stereoscopic head-mounted sound system, head-motion trackingsensors (such as gyroscopes, accelerometers, magnetometers, imagesensors, structured light sensors, etc.), head mounted projectors,eye-tracking sensors, and additional components described below.Consistent with another aspect of the disclosure, the extended realityappliance may be a non-wearable extended reality appliance.Specifically, the non-wearable extended reality appliance may includemulti-projected environment appliances. In some embodiments, an extendedreality appliance may be configured to change the viewing perspective ofthe extended reality environment in response to movements of the userand in response to head movements of the user in particular. In oneexample, a wearable extended reality appliance may change thefield-of-view of the extended reality environment in response to achange of the head pose of the user, such as by changing the spatialorientation without changing the spatial position of the user in theextended reality environment. In another example, a non-wearableextended reality appliance may change the spatial position of the userin the extended reality environment in response to a change in theposition of the user in the real world, for example, by changing thespatial position of the user in the extended reality environment withoutchanging the direction of the field-of-view with respect to the spatialposition.

According to some embodiments, an extended reality appliance may includea digital communication device configured to at least one of: receivingvirtual content data configured to enable a presentation of the virtualcontent, transmitting virtual content for sharing with at least oneexternal device, receiving contextual data from at least one externaldevice, transmitting contextual data to at least one external device,transmitting of usage data indicative of usage of the extended realityappliance, and transmitting of data based on information captured usingat least one sensor included in the extended reality appliance. Inadditional embodiments, the extended reality appliance may includememory for storing at least one of virtual data configured to enable apresentation of virtual content, contextual data, usage data indicativeof usage of the extended reality appliance, sensor data based oninformation captured using at least one sensor included in the extendedreality appliance, software instructions configured to cause aprocessing device to present the virtual content, software instructionsconfigured to cause a processing device to collect and analyze thecontextual data, software instructions configured to cause a processingdevice to collect and analyze the usage data, and software instructionsconfigured to cause a processing device to collect and analyze thesensor data. In additional embodiments, the extended reality appliancemay include a processing device configured to perform at least one ofrendering of virtual content, collecting and analyzing contextual data,collecting and analyzing usage data, and collecting and analyzing sensordata. In additional embodiments, the extended reality appliance mayinclude one or more sensors. The one or more sensors may include one ormore image sensors (e.g., configured to capture images and/or videos ofa user of the appliance or of an environment of the user), one or moremotion sensors (such as an accelerometer, a gyroscope, a magnetometer,etc.), one or more positioning sensors (such as GPS, outdoor positioningsensor, indoor positioning sensor, etc.), one or more temperaturesensors (e.g., configured to measure the temperature of at least part ofthe appliance and/or of the environment), one or more contact sensors,one or more proximity sensors (e.g., configured to detect whether theappliance is currently worn), one or more electrical impedance sensors(e.g., configured to measure electrical impedance of the user), one ormore eye tracking sensors, such as gaze detectors, optical trackers,electric potential trackers (e.g., electrooculogram (EOG) sensors),video-based eye-trackers, infra-red/near infra-red sensors, passivelight sensors, or any other technology capable of determining where ahuman is looking or gazing.

In some embodiments, the systems and methods may use an input device tointeract with the extended reality appliance. The term input device mayinclude any physical device configured to receive input from a user oran environment of the user, and to provide the data to a computationaldevice. The data provided to the computational device may be in adigital format and/or in an analog format. In one embodiment, the inputdevice may store the input received from the user in a memory deviceaccessible by a processing device, and the processing device may accessthe stored data for analysis. In another embodiment, the input devicemay provide the data directly to a processing device, for example, overa bus or over another communication system configured to transfer datafrom the input device to the processing device. In some examples, theinput received by the input device may include key presses, tactileinput data, motion data, position data, gestures based input data,direction data, or any other data for supply for computation. Someexamples of the input device may include a button, a key, a keyboard, acomputer mouse, a touchpad, a touchscreen, a joystick, or anothermechanism from which input may be received. Another example of an inputdevice may include an integrated computational interface device thatincludes at least one physical component for receiving input from auser. The integrated computational interface device may include at leasta memory, a processing device, and the at least one physical componentfor receiving input from a user. In one example, the integratedcomputational interface device may further include a digital networkinterface that enables digital communication with other computingdevices. In one example, the integrated computational interface devicemay further include a physical component for outputting information tothe user. In some examples, all components of the integratedcomputational interface device may be included in a single housing,while in other examples the components may be distributed among two ormore housings. Some non-limiting examples of physical components forreceiving input from users that may be included in the integratedcomputational interface device may include at least one of a button, akey, a keyboard, a touchpad, a touchscreen, a joystick, or any othermechanism or sensor from which computational information may bereceived. Some non-limiting examples of physical components foroutputting information to users may include at least one of a lightindicator (such as a LED indicator), a screen, a touchscreen, a beeper,an audio speaker, or any other audio, video, or haptic device thatprovides human-perceptible outputs.

In some embodiments, image data may be captured using one or more imagesensors. In some examples, the image sensors may be included in theextended reality appliance, in a wearable device, in the wearableextended reality device, in the input device, in an environment of auser, and so forth. In some examples, the image data may be read frommemory, may be received from an external device, may be generated (forexample, using a generative model), and so forth. Some non-limitingexamples of image data may include images, grayscale images, colorimages, 2D images, 3D images, videos, 2D videos, 3D videos, frames,footages, data derived from other image data, and so forth. In someexamples, the image data may be encoded in any analog or digital format.Some non-limiting examples of such formats may include raw formats,compressed formats, uncompressed formats, lossy formats, losslessformats, JPEG, GIF, PNG, TIFF, BMP, NTSC, PAL, SECAM, MPEG, MPEG-4 Part14, MOV, WMV, FLV, AVI, AVCHD, WebM, MKV, and so forth.

In some embodiments, the extended reality appliance may receive digitalsignals, for example, from the input device. The term digital signalsrefers to a series of digital values that are discrete in time. Thedigital signals may represent, for example, sensor data, textual data,voice data, video data, virtual data, or any other form of data thatprovides perceptible information. Consistent with the presentdisclosure, the digital signals may be configured to cause the extendedreality appliance to present virtual content. In one embodiment, thevirtual content may be presented in a selected orientation. In thisembodiment, the digital signals may indicate a position and an angle ofa viewpoint in an environment, such as an extended reality environment.Specifically, the digital signals may include an encoding of theposition and angle in six degree-of-freedom coordinates (e.g.,forward/back, up/down, left/right, yaw, pitch, and roll). In anotherembodiment, the digital signals may include an encoding of the positionas three-dimensional coordinates (e.g., x, y, and z), and an encoding ofthe angle as a vector originating from the encoded position.Specifically, the digital signals may indicate the orientation and anangle of the presented virtual content in an absolute coordinates of theenvironment, for example, by encoding yaw, pitch and roll of the virtualcontent with respect to a standard default angle. In another embodiment,the digital signals may indicate the orientation and the angle of thepresented virtual content with respect to a viewpoint of another object(e.g., a virtual object, a physical object, etc.), for example, byencoding yaw, pitch, and roll of the virtual content with respect adirection corresponding to the viewpoint or to a direction correspondingto the other object. In another embodiment, such digital signals mayinclude one or more projections of the virtual content, for example, ina format ready for presentation (e.g., image, video, etc.). For example,each such projection may correspond to a particular orientation or aparticular angle. In another embodiment, the digital signals may includea representation of virtual content, for example, by encoding objects ina three-dimensional array of voxels, in a polygon mesh, or in any otherformat in which virtual content may be presented.

In some embodiments, the digital signals may be configured to cause theextended reality appliance to present virtual content. The term virtualcontent may include any type of data representation that may bedisplayed by the extended reality appliance to the user. The virtualcontent may include a virtual object, inanimate virtual content, animatevirtual content configured to change over time or in response totriggers, virtual two-dimensional content, virtual three dimensionalcontent, a virtual overlay over a portion of a physical environment orover a physical object, a virtual addition to a physical environment orto a physical object, a virtual promotion content, a virtualrepresentation of a physical object, a virtual representation of aphysical environment, a virtual document, a virtual character orpersona, a virtual computer screen, a virtual widget, or any otherformat for displaying information virtually. Consistent with the presentdisclosure, the virtual content may include any visual presentationrendered by a computer or a processing device. In one embodiment, thevirtual content may include a virtual object that is a visualpresentation rendered by a computer in a confined region and configuredto represent an object of a particular type (such as an inanimatevirtual object, an animate virtual object, virtual furniture, a virtualdecorative object, virtual widget, or other virtual representation). Therendered visual presentation may change to reflect changes to a statusobject or changes in the viewing angle of the object, for example, in away that mimics changes in the appearance of physical objects. Inanother embodiment, the virtual content may include a virtual display(also referred to as a “virtual display screen” or a “virtual screen”herein), such as a virtual computer screen, a virtual tablet screen or avirtual smartphone screen, configured to display information generatedby an operating system, in which the operating system may be configuredto receive textual data from a physical keyboard and/or a virtualkeyboard and to cause a display of the textual content in the virtualdisplay screen. In one example, illustrated in FIG. 1, the virtualcontent may include a virtual environment that includes a virtualcomputer screen and a plurality of virtual objects. In some examples, avirtual display may be a virtual object mimicking and/or extending thefunctionality of a physical display screen. For example, the virtualdisplay may be presented in an extended reality environment (such as amixed reality environment, an augmented reality environment, a virtualreality environment, etc.), using an extended reality appliance. In oneexample, a virtual display may present content produced by a regularoperating system that may be equally presented on a physical displayscreen. In one example, a textual content entered using a keyboard (forexample, using a physical keyboard, using a virtual keyboard, etc.) maybe presented on a virtual display in real time as the textual content istyped. In one example, a virtual cursor may be presented on a virtualdisplay, and the virtual cursor may be controlled by a pointing device(such as a physical pointing device, a virtual pointing device, acomputer mouse, a joystick, a touchpad, a physical touch controller, andso forth). In one example, one or more windows of a graphical userinterface operating system may be presented on a virtual display. Inanother example, content presented on a virtual display may beinteractive, that is, it may change in reaction to actions of users. Inyet another example, a presentation of a virtual display may include apresentation of a screen frame, or may include no presentation of ascreen frame.

Some disclosed embodiments may include and/or access a data structure ora database. The terms data structure and a database, consistent with thepresent disclosure may include any collection of data values andrelationships among them. The data may be stored linearly, horizontally,hierarchically, relationally, non-relationally, uni-dimensionally,multidimensionally, operationally, in an ordered manner, in an unorderedmanner, in an object-oriented manner, in a centralized manner, in adecentralized manner, in a distributed manner, in a custom manner, or inany manner enabling data access. By way of non-limiting examples, datastructures may include an array, an associative array, a linked list, abinary tree, a balanced tree, a heap, a stack, a queue, a set, a hashtable, a record, a tagged union, Entity-Relationship model, a graph, ahypergraph, a matrix, a tensor, and so forth. For example, a datastructure may include an XML database, an RDBMS database, an SQLdatabase or NoSQL alternatives for data storage/search such as, forexample, MongoDB, Redis, Couchbase, Datastax Enterprise Graph, ElasticSearch, Splunk, Solr, Cassandra, Amazon DynamoDB, Scylla, HBase, andNeo4J. A data structure may be a component of the disclosed system or aremote computing component (e.g., a cloud-based data structure). Data inthe data structure may be stored in contiguous or non-contiguous memory.Moreover, a data structure does not require information to beco-located. It may be distributed across multiple servers, for example,that may be owned or operated by the same or different entities. Thus,the term data structure in the singular is inclusive of plural datastructures.

In some embodiments, the system may determine the confidence level inreceived input or in any determined value. The term confidence levelrefers to any indication, numeric or otherwise, of a level (e.g., withina predetermined range) indicative of an amount of confidence the systemhas at determined data. For example, the confidence level may have avalue between 1 and 10. Alternatively, the confidence level may beexpressed as a percentage or any other numerical or non-numericalindication. In some cases, the system may compare the confidence levelto a threshold. The term threshold may denote a reference value, alevel, a point, or a range of values. In operation, when the confidencelevel of determined data exceeds the threshold (or is below it,depending on a particular use case), the system may follow a firstcourse of action and, when the confidence level is below it (or aboveit, depending on a particular use case), the system may follow a secondcourse of action. The value of the threshold may be predetermined foreach type of examined object or may be dynamically selected based ondifferent considerations.

System Overview

Reference is now made to FIG. 1, which illustrates a user that uses anexample extended reality system consistent with various embodiments ofthe present disclosure. FIG. 1 is an exemplary representation of justone embodiment, and it is to be understood that some illustratedelements might be omitted and others added within the scope of thisdisclosure. As shown, a user 100 is sitting behind table 102, supportinga keyboard 104 and mouse 106. Keyboard 104 is connected by wire 108 to awearable extended reality appliance 110 that displays virtual content touser 100. Alternatively or additionally to wire 108, keyboard 104 mayconnect to wearable extended reality appliance 110 wirelessly. Forillustration purposes, the wearable extended reality appliance isdepicted as a pair of smart glasses, but, as described above, wearableextended reality appliance 110 may be any type of head-mounted deviceused for presenting an extended reality to user 100. The virtual contentdisplayed by wearable extended reality appliance 110 includes a virtualscreen 112 (also referred to as a “virtual display screen” or a “virtualdisplay” herein) and a plurality of virtual widgets 114. Virtual widgets114A-114D are displayed next to virtual screen 112 and virtual widget114E is displayed on table 102. User 100 may input text to a document116 displayed in virtual screen 112 using keyboard 104; and may controlvirtual cursor 118 using mouse 106. In one example, virtual cursor 118may move anywhere within virtual screen 112. In another example, virtualcursor 118 may move anywhere within virtual screen 112 and may also moveto any one of virtual widgets 114A-114D but not to virtual widget 114E.In yet another example, virtual cursor 118 may move anywhere withinvirtual screen 112 and may also move to any one of virtual widgets114A-114E. In an additional example, virtual cursor 118 may moveanywhere in the extended reality environment including virtual screen112 and virtual widgets 114A-114E. In yet another example, virtualcursor may move on all available surfaces (i.e., virtual surfaces orphysical surfaces) or only on selected surfaces in the extended realityenvironment. Alternatively or additionally, user 100 may interact withany one of virtual widgets 114A-114E, or with selected virtual widgets,using hand gestures recognized by wearable extended reality appliance110. For example, virtual widget 114E may be an interactive widget(e.g., a virtual slider controller) that may be operated with handgestures.

FIG. 2 illustrates an example of a system 200 that provides extendedreality (XR) experience to users, such as user 100. FIG. 2 is anexemplary representation of just one embodiment, and it is to beunderstood that some illustrated elements might be omitted and othersadded within the scope of this disclosure. System 200 may becomputer-based and may include computer system components, wearableappliances, workstations, tablets, handheld computing devices, memorydevices, and/or internal network(s) connecting the components. System200 may include or be connected to various network computing resources(e.g., servers, routers, switches, network connections, storage devices,etc.) for supporting services provided by system 200. Consistent withthe present disclosure, system 200 may include an input unit 202, an XRunit 204, a mobile communications device 206, and a remote processingunit 208. Remote processing unit 208 may include a server 210 coupled toone or more physical or virtual storage devices, such as a datastructure 212. System 200 may also include or be connected to acommunications network 214 that facilitates communications and dataexchange between different system components and the different entitiesassociated with system 200.

Consistent with the present disclosure, input unit 202 may include oneor more devices that may receive input from user 100. In one embodiment,input unit 202 may include a textual input device, such as keyboard 104.The textual input device may include all possible types of devices andmechanisms for inputting textual information to system 200. Examples oftextual input devices may include mechanical keyboards, membranekeyboards, flexible keyboards, QWERTY keyboards, Dvorak keyboards,Colemak keyboards, chorded keyboards, wireless keyboards, keypads,key-based control panels, or other arrays of control keys, vision inputdevices, or any other mechanism for inputting text, whether themechanism is provided in physical form or is presented virtually. In oneembodiment, input unit 202 may also include a pointing input device,such as mouse 106. The pointing input device may include all possibletypes of devices and mechanisms for inputting two-dimensional orthree-dimensional information to system 200. In one example,two-dimensional input from the pointing input device may be used forinteracting with virtual content presented via the XR unit 204. Examplesof pointing input devices may include a computer mouse, trackball,touchpad, trackpad, touchscreen, joystick, pointing stick, stylus, lightpen, or any other physical or virtual input mechanism. In oneembodiment, input unit 202 may also include a graphical input device,such as a touchscreen configured to detect contact, movement, or breakof movement. The graphical input device may use any of a plurality oftouch sensitivity technologies, including, but not limited to,capacitive, resistive, infrared, and surface acoustic wave technologiesas well as other proximity sensor arrays or other elements fordetermining one or more points of contact. In one embodiment, input unit202 may also include one or more voice input devices, such as amicrophone. The voice input device may include all possible types ofdevices and mechanisms for inputting voice data to facilitatevoice-enabled functions, such as voice recognition, voice replication,digital recording, and telephony functions. In one embodiment, inputunit 202 may also include one or more image input devices, such as animage sensor, configured to capture image data. In one embodiment, inputunit 202 may also include one or more haptic gloves configured tocapture hands motion and pose data. In one embodiment, input unit 202may also include one or more proximity sensors configured to detectpresence and/or movement of objects in a selected region near thesensors.

In accordance with some embodiments, the system may include at least onesensor configured to detect and/or measure a property associated withthe user, the user's action, or user's environment. One example of theat least one sensor, is sensor 216 included in input unit 202. Sensor216 may be a motion sensor, a touch sensor, a light sensor, an infraredsensor, an audio sensor, an image sensor, a proximity sensor, apositioning sensor, a gyroscope, a temperature sensor, a biometricsensor, or any other sensing devices to facilitate relatedfunctionalities. Sensor 216 may be integrated with, or connected to, theinput devices or it may be separated from the input devices. In oneexample, a thermometer may be included in mouse 106 to determine thebody temperature of user 100. In another example, a positioning sensormay be integrated with keyboard 104 to determine movement of user 100relative to keyboard 104. Such positioning sensor may be implementedusing one of the following technologies: Global Positioning System(GPS), GLObal NAvigation Satellite System (GLONASS), Galileo globalnavigation system, BeiDou navigation system, other Global NavigationSatellite Systems (GNSS), Indian Regional Navigation Satellite System(IRNSS), Local Positioning Systems (LPS), Real-Time Location Systems(RTLS), Indoor Positioning System (IPS), Wi-Fi based positioningsystems, cellular triangulation, image based positioning technology,indoor positioning technology, outdoor positioning technology, or anyother positioning technology.

In accordance with some embodiments, the system may include one or moresensors for identifying a position and/or a movement of a physicaldevice (such as a physical input device, a physical computing device,keyboard 104, mouse 106, wearable extended reality appliance 110, and soforth). The one or more sensors may be included in the physical deviceor may be external to the physical device. In some examples, an imagesensor external to the physical device (for example, an image sensorincluded in another physical device) may be used to capture image dataof the physical device, and the image data may be analyzed to identifythe position and/or the movement of the physical device. For example,the image data may be analyzed using a visual object tracking algorithmto identify the movement of the physical device, may be analyzed using avisual object detection algorithm to identify the position of thephysical device (for example, relative to the image sensor, in a globalcoordinates system, etc.), and so forth. In some examples, an imagesensor included in the physical device may be used to capture imagedata, and the image data may be analyzed to identify the position and/orthe movement of the physical device. For example, the image data may beanalyzed using visual odometry algorithms to identify the position ofthe physical device, may be analyzed using an ego-motion algorithm toidentify movement of the physical device, and so forth. In someexamples, a positioning sensor, such as an indoor positioning sensor oran outdoor positioning sensor, may be included in the physical deviceand may be used to determine the position of the physical device. Insome examples, a motion sensor, such as an accelerometer or a gyroscope,may be included in the physical device and may be used to determine themotion of the physical device. In some examples, a physical device, suchas a keyboard or a mouse, may be configured to be positioned on aphysical surface. Such physical device may include an optical mousesensor (also known as non-mechanical tracking engine) aimed towards thephysical surface, and the output of the optical mouse sensor may beanalyzed to determine movement of the physical device with respect tothe physical surface.

Consistent with the present disclosure, XR unit 204 may include awearable extended reality appliance configured to present virtualcontent to user 100. One example of the wearable extended realityappliance is wearable extended reality appliance 110. Additionalexamples of wearable extended reality appliance may include a VirtualReality (VR) device, an Augmented Reality (AR) device, a Mixed Reality(MR) device, or any other device capable of generating extended realitycontent. Some non-limiting examples of such devices may include NrealLight, Magic Leap One, Varjo, Quest 1/2, Vive, and others. In someembodiments, XR unit 204 may present virtual content to user 100.Generally, an extended reality appliance may include all real-and-virtual combined environments and human-machine interactionsgenerated by computer technology and wearables. As mentioned above, theterm “extended reality” (XR) refers to a superset which includes theentire spectrum from “the complete real” to “the complete virtual.” Itincludes representative forms such as augmented reality (AR), mixedreality (MR), virtual reality (VR), and the areas interpolated amongthem. Accordingly, it is noted that the terms “XR appliance,” “ARappliance,” “VR appliance,” and “MR appliance” may be usedinterchangeably herein and may refer to any device of the variety ofappliances listed above.

Consistent with the present disclosure, the system may exchange datawith a variety of communication devices associated with users, forexample, mobile communications device 206. The term “communicationdevice” is intended to include all possible types of devices capable ofexchanging data using digital communications network, analogcommunication network or any other communications network configured toconvey data. In some examples, the communication device may include asmartphone, a tablet, a smartwatch, a personal digital assistant, adesktop computer, a laptop computer, an IoT device, a dedicatedterminal, a wearable communication device, and any other device thatenables data communications. In some cases, mobile communications device206 may supplement or replace input unit 202. Specifically, mobilecommunications device 206 may be associated with a physical touchcontroller that may function as a pointing input device. Moreover,mobile communications device 206 may also, for example, be used toimplement a virtual keyboard and replace the textual input device. Forexample, when user 100 steps away from table 102 and walks to the breakroom with his smart glasses, he may receive an email that requires aquick answer. In this case, the user may select to use his or her ownsmartwatch as the input device and to type the answer to the email whileit is virtually presented by the smart glasses.

Consistent with the present disclosure, embodiments of the system mayinvolve the usage of a cloud server. The term “cloud server” refers to acomputer platform that provides services via a network, such as theInternet. In the example embodiment illustrated in FIG. 2, server 210may use virtual machines that may not correspond to individual hardware.For example, computational and/or storage capabilities may beimplemented by allocating appropriate portions of desirablecomputation/storage power from a scalable repository, such as a datacenter or a distributed computing environment. Specifically, in oneembodiment, remote processing unit 208 may be used together with XR unit204 to provide the virtual content to user 100. In one exampleconfiguration, server 210 may be a cloud server that functions as theoperation system (OS) of the wearable extended reality appliance. In oneexample, server 210 may implement the methods described herein usingcustomized hard-wired logic, one or more Application Specific IntegratedCircuits (ASICs), Field Programmable Gate Arrays (FPGAs), firmware,and/or program logic which, in combination with the computer system,cause server 210 to be a special-purpose machine.

In some embodiments, server 210 may access data structure 212 todetermine, for example, virtual content to display user 100. Datastructure 212 may utilize a volatile or non-volatile, magnetic,semiconductor, tape, optical, removable, non-removable, other type ofstorage device or tangible or non-transitory computer-readable medium,or any medium or mechanism for storing information. Data structure 212may be part of server 210 or separate from server 210, as shown. Whendata structure 212 is not part of server 210, server 210 may exchangedata with data structure 212 via a communication link. Data structure212 may include one or more memory devices that store data andinstructions used to perform one or more features of the disclosedmethods. In one embodiment, data structure 212 may include any of aplurality of suitable data structures, ranging from small datastructures hosted on a workstation to large data structures distributedamong data centers. Data structure 212 may also include any combinationof one or more data structures controlled by memory controller devices(e.g., servers) or software.

Consistent with the present disclosure, communications network may beany type of network (including infrastructure) that supportscommunications, exchanges information, and/or facilitates the exchangeof information between the components of a system. For example,communications network 214 in system 200 may include, for example, atelephone network, an extranet, an intranet, the Internet, satellitecommunications, off-line communications, wireless communications,transponder communications, a Local Area Network (LAN), wireless network(e.g., a Wi-Fi/302.11 network), a Wide Area Network (WAN), a VirtualPrivate Network (VPN), digital communication network, analogcommunication network, or any other mechanism or combinations ofmechanism that enable data transmission.

The components and arrangements of system 200 shown in FIG. 2 areintended to be exemplary only and are not intended to limit anyembodiment, as the system components used to implement the disclosedprocesses and features may vary.

FIG. 3 is a block diagram of an examplary configuration of input unit202. FIG. 3 is an exemplary representation of just one embodiment, andit is to be understood that some illustrated elements might be omittedand others added within the scope of this disclosure. In the embodimentof FIG. 3, input unit 202 may directly or indirectly access a bus 300(or other communication mechanism) that interconnects subsystems andcomponents for transferring information within input unit 202. Forexample, bus 300 may interconnect a memory interface 310, a networkinterface 320, an input interface 330, a power source 340, an outputinterface 350, a processing device 360, a sensors interface 370, and adatabase 380.

Memory interface 310, shown in FIG. 3, may be used to access a softwareproduct and/or data stored on a non-transitory computer-readable medium.Generally, a non-transitory computer-readable storage medium refers toany type of physical memory on which information or data readable by atleast one processor can be stored. Examples include Random Access Memory(RAM), Read-Only Memory (ROM), volatile memory, nonvolatile memory, harddrives, CD ROMs, DVDs, flash drives, disks, any other optical datastorage medium, any physical medium with patterns of holes, a PROM, anEPROM, a FLASH-EPROM or any other flash memory, NVRAM, a cache, aregister, any other memory chip or cartridge, and networked versions ofthe same. The terms “memory” and “computer-readable storage medium” mayrefer to multiple structures, such as a plurality of memories orcomputer-readable storage mediums located within an input unit or at aremote location. Additionally, one or more computer-readable storagemediums can be utilized in implementing a computer-implemented method.Accordingly, the term computer-readable storage medium should beunderstood to include tangible items and exclude carrier waves andtransient signals. In the specific embodiment illustrated in FIG. 3,memory interface 310 may be used to access a software product and/ordata stored on a memory device, such as memory device 311. Memory device311 may include high-speed random-access memory and/or non-volatilememory, such as one or more magnetic disk storage devices, one or moreoptical storage devices, and/or flash memory (e.g., NAND, NOR).Consistent with the present disclosure, the components of memory device311 may be distributed in more than units of system 200 and/or in morethan one memory device.

Memory device 311, shown in FIG. 3, may contain software modules toexecute processes consistent with the present disclosure. In particular,memory device 311 may include an input determination module 312, anoutput determination module 313, a sensors communication module 314, avirtual content determination module 315, a virtual contentcommunication module 316, and a database access module 317. Modules312-317 may contain software instructions for execution by at least oneprocessor (e.g., processing device 360) associated with input unit 202.Input determination module 312, output determination module 313, sensorscommunication module 314, virtual content determination module 315,virtual content communication module 316, and database access module 317may cooperate to perform various operations. For example, inputdetermination module 312 may determine text using data received from,for example, keyboard 104. Thereafter, output determination module 313may cause presentation of the recent inputted text, for example on adedicated display 352 physically or wirelessly coupled to keyboard 104.This way, when user 100 types, he can see a preview of the typed textwithout constantly moving his head up and down to look at virtual screen112. Sensors communication module 314 may receive data from differentsensors to determine a status of user 100. Thereafter, virtual contentdetermination module 315 may determine the virtual content to display,based on received input and the determined status of user 100. Forexample, the determined virtual content may be a virtual presentation ofthe recent inputted text on a virtual screen virtually located adjacentto keyboard 104. Virtual content communication module 316 may obtainvirtual content that is not determined by virtual content determinationmodule 315 (e.g., an avatar of another user). The retrieval of thevirtual content may be from database 380, from remote processing unit208, or any other source.

In some embodiments, input determination module 312 may regulate theoperation of input interface 330 in order to receive pointer input 331,textual input 332, audio input 333, and XR-related input 334. Details onthe pointer input, the textual input, and the audio input are describedabove. The term “XR-related input” may include any type of data that maycause a change in the virtual content displayed to user 100. In oneembodiment, XR-related input 334 may include image data of user 100, awearable extended reality appliance (e.g., detected hand gestures ofuser 100). In another embodiment, XR-related input 334 may includewireless communication indicating a presence of another user inproximity to user 100. Consistent with the present disclosure, inputdetermination module 312 may concurrently receive different types ofinput data. Thereafter, input determination module 312 may further applydifferent rules based on the detected type of input. For example, apointer input may have precedence over voice input.

In some embodiments, output determination module 313 may regulate theoperation of output interface 350 in order to generate output usinglight indicators 351, display 352, and/or speakers 353. In general, theoutput generated by output determination module 313 does not includevirtual content to be presented by a wearable extended realityappliance. Instead, the output generated by output determination module313 include various outputs that relates to the operation of input unit202 and/or the operation of XR unit 204. In one embodiment, lightindicators 351 may include a light indicator that shows the status of awearable extended reality appliance. For example, the light indicatormay display green light when wearable extended reality appliance 110 areconnected to keyboard 104, and blinks when wearable extended realityappliance 110 has low battery. In another embodiment, display 352 may beused to display operational information. For example, the display maypresent error messages when the wearable extended reality appliance isinoperable. In another embodiment, speakers 353 may be used to outputaudio, for example, when user 100 wishes to play some music for otherusers.

In some embodiments, sensors communication module 314 may regulate theoperation of sensors interface 370 in order to receive sensor data fromone or more sensors, integrated with, or connected to, an input device.The one or more sensors may include: audio sensor 371, image sensor 372,motion sensor 373, environmental sensor 374 (e.g., a temperature sensor,ambient light detectors, etc.), and other sensors 375. In oneembodiment, the data received from sensors communication module 314 maybe used to determine the physical orientation of the input device. Thephysical orientation of the input device may be indicative of a state ofthe user and may be determined based on combination of a tilt movement,a roll movement, and a lateral movement. Thereafter, the physicalorientation of the input device may be used by virtual contentdetermination module 315 to modify display parameters of the virtualcontent to match the state of the user (e.g., attention, sleepy, active,sitting, standing, leaning backwards, leaning forward, walking, moving,riding, etc.).

In some embodiments, virtual content determination module 315 maydetermine the virtual content to be displayed by the wearable extendedreality appliance. The virtual content may be determined based on datafrom input determination module 312, sensors communication module 314,and other sources (e.g., database 380). In some embodiments, determiningthe virtual content may include determining the distance, the size, andthe orientation of the virtual objects. The determination of theposition of the virtual objects may be determined based on the type ofthe virtual objects. Specifically, with regards to the exampleillustrated in FIG. 1, the virtual content determination module 315 maydetermine to place four virtual widgets 114A-114D on the sides ofvirtual screen 112 and to place virtual widget 114E on table 102 becausevirtual widget 114E is a virtual controller (e.g., volume bar). Thedetermination of the position of the virtual objects may further bedetermined based on user's preferences. For example, for left-handedusers, virtual content determination module 315 may determine placing avirtual volume bar left of keyboard 104; and for right-handed users,virtual content determination module 315 may determine placing thevirtual volume bar right of keyboard 104.

In some embodiments, virtual content communication module 316 mayregulate the operation of network interface 320 in order to obtain datafrom one or more sources to be presented as virtual content to user 100.The one or more sources may include other XR units 204, the user'smobile communications device 206, remote processing unit 208, publiclyavailable information, etc. In one embodiment, virtual contentcommunication module 316 may communicate with mobile communicationsdevice 206 in order to provide a virtual representation of mobilecommunications device 206. For example, the virtual representation mayenable user 100 to read messages and interact with applicationsinstalled on the mobile communications device 206. Virtual contentcommunication module 316 may also regulate the operation of networkinterface 320 in order to share virtual content with other users. In oneexample, virtual content communication module 316 may use data frominput determination module to identify a trigger (e.g., the trigger mayinclude a gesture of the user) and to transfer content from the virtualdisplay to a physical display (e.g., TV) or to a virtual display of adifferent user.

In some embodiments, database access module 317 may cooperate withdatabase 380 to retrieve stored data. The retrieved data may include,for example, privacy levels associated with different virtual objects,the relationship between virtual objects and physical objects, theuser's preferences, the user's past behavior, and more. As describedabove, virtual content determination module 315 may use the data storedin database 380 to determine the virtual content. Database 380 mayinclude separate databases, including, for example, a vector database,raster database, tile database, viewport database, and/or a user inputdatabase. The data stored in database 380 may be received from modules314-317 or other components of system 200. Moreover, the data stored indatabase 380 may be provided as input using data entry, data transfer,or data uploading.

Modules 312-317 may be implemented in software, hardware, firmware, amix of any of those, or the like. In some embodiments, any one or moreof modules 312-317 and data associated with database 380 may be storedin XR unit 204, mobile communications device 206, or remote processingunit 208. Processing devices of system 200 may be configured to executethe instructions of modules 312-317. In some embodiments, aspects ofmodules 312-317 may be implemented in hardware, in software (includingin one or more signal processing and/or application specific integratedcircuits), in firmware, or in any combination thereof, executable by oneor more processors, alone, or in various combinations with each other.Specifically, modules 312-317 may be configured to interact with eachother and/or other modules of system 200 to perform functions consistentwith Some disclosed embodiments. For example, input unit 202 may executeinstructions that include an image processing algorithm on data from XRunit 204 to determine head movement of user 100. Furthermore, eachfunctionality described throughout the specification, with regards toinput unit 202 or with regards to a component of input unit 202, maycorrespond to a set of instructions for performing said functionality.These instructions need not be implemented as separate softwareprograms, procedures, or modules. Memory device 311 may includeadditional modules and instructions or fewer modules and instructions.For example, memory device 311 may store an operating system, such asANDROID, iOS, UNIX, OSX, WINDOWS, DARWIN, RTXC, LINUX or an embeddedoperating system such as VXWorkS. The operating system can includeinstructions for handling basic system services and for performinghardware-dependent tasks.

Network interface 320, shown in FIG. 3, may provide two-way datacommunications to a network, such as communications network 214. In oneembodiment, network interface 320 may include an Integrated ServicesDigital Network (ISDN) card, cellular modem, satellite modem, or a modemto provide a data communication connection over the Internet. As anotherexample, network interface 320 may include a Wireless Local Area Network(WLAN) card. In another embodiment, network interface 320 may include anEthernet port connected to radio frequency receivers and transmittersand/or optical (e.g., infrared) receivers and transmitters. The specificdesign and implementation of network interface 320 may depend on thecommunications network or networks over which input unit 202 is intendedto operate. For example, in some embodiments, input unit 202 may includenetwork interface 320 designed to operate over a GSM network, a GPRSnetwork, an EDGE network, a Wi-Fi or WiMax network, and a Bluetoothnetwork. In any such implementation, network interface 320 may beconfigured to send and receive electrical, electromagnetic, or opticalsignals that carry digital data streams or digital signals representingvarious types of information.

Input interface 330, shown in FIG. 3, may receive input from a varietyof input devices, for example, a keyboard, a mouse, a touch pad, a touchscreen, one or more buttons, a joystick, a microphone, an image sensor,and any other device configured to detect physical or virtual input. Thereceived input may be in the form of at least one of: text, sounds,speech, hand gestures, body gestures, tactile information, and any othertype of physically or virtually input generated by the user. In thedepicted embodiment, input interface 330 may receive pointer input 331,textual input 332, audio input 333, and XR-related input 334. Inadditional embodiment, input interface 330 may be an integrated circuitthat may act as bridge between processing device 360 and any of theinput devices listed above.

Power source 340, shown in FIG. 3, may provide electrical energy topower input unit 202 and optionally also power XR unit 204. Generally, apower source included in the any device or system in the presentdisclosure may be any device that can repeatedly store, dispense, orconvey electric power, including, but not limited to, one or morebatteries (e.g., a lead-acid battery, a lithium-ion battery, anickel-metal hydride battery, a nickel-cadmium battery), one or morecapacitors, one or more connections to external power sources, one ormore power convertors, or any combination of them. With reference to theexample illustrated in FIG. 3, the power source may be mobile, whichmeans that input unit 202 can be easily carried by a hand (e.g., thetotal weight of power source 340 may be less than a pound). The mobilityof the power source enables user 100 to use input unit 202 in a varietyof situations. In other embodiments, power source 340 may be associatedwith a connection to an external power source (such as an electricalpower grid) that may be used to charge power source 340. In addition,power source 340 may be configured to charge one or more batteriesincluded in XR unit 204; for example, a pair of extended reality glasses(e.g., wearable extended reality appliance 110) may be charged (e.g.,wirelessly or not wirelessly) when they are placed on or in proximity tothe input unit 202.

Output interface 350, shown in FIG. 3, may cause output from a varietyof output devices, for example, using light indicators 351, display 352,and/or speakers 353. In one embodiment, output interface 350 may be anintegrated circuit that may act as bridge between processing device 360and at least one of the output devices listed above. Light indicators351 may include one or more light sources, for example, a LED arrayassociated with different colors. Display 352 may include a screen(e.g., LCD or dot-matrix screen) or a touch screen. Speakers 353 mayinclude audio headphones, a hearing aid type device, a speaker, a boneconduction headphone, interfaces that provide tactile cues, vibrotactilestimulators, and more.

Processing device 360, shown in FIG. 3, may include at least oneprocessor configured to execute computer programs, applications,methods, processes, or other software to perform embodiments describedin the present disclosure. Generally, a processing device included inthe any device or system in the present disclosure may include one ormore integrated circuits, microchips, microcontrollers, microprocessors,all or part of a central processing unit (CPU), graphics processing unit(GPU), digital signal processor (DSP), field programmable gate array(FPGA), or other circuits suitable for executing instructions orperforming logic operations. The processing device may include at leastone processor configured to perform functions of the disclosed methodssuch as a microprocessor manufactured by Intel™. The processing devicemay include a single core or multiple core processors executing parallelprocesses simultaneously. In one example, the processing device may be asingle core processor configured with virtual processing technologies.The processing device may implement virtual machine technologies orother technologies to provide the ability to execute, control, run,manipulate, store, etc., multiple software processes, applications,programs, etc. In another example, the processing device may include amultiple-core processor arrangement (e.g., dual, quad core, etc.)configured to provide parallel processing functionalities to allow adevice associated with the processing device to execute multipleprocesses simultaneously. It is appreciated that other types ofprocessor arrangements could be implemented to provide the capabilitiesdisclosed herein.

Sensors interface 370, shown in FIG. 3, may obtain sensor data from avariety of sensors, for example, audio sensor 371, image sensor 372,motion sensor 373, environmental sensor 374, and other sensors 375. Inone embodiment, sensors interface 370 may be an integrated circuit thatmay act as bridge between processing device 360 and at least one of thesensors listed above.

Audio sensor 371 may include one or more audio sensors configured tocapture audio by converting sounds to digital information. Some examplesof audio sensors may include: microphones, unidirectional microphones,bidirectional microphones, cardioid microphones, omnidirectionalmicrophones, onboard microphones, wired microphones, wirelessmicrophones, or any combination of the above. Consistent with thepresent disclosure, processing device 360 may modify a presentation ofvirtual content based on data received from audio sensor 371 (e.g.,voice commands).

Image sensor 372 may include one or more image sensors configured tocapture visual information by converting light to image data. Consistentwith the present disclosure, an image sensor may be included in the anydevice or system in the present disclosure and may be any device capableof detecting and converting optical signals in the near-infrared,infrared, visible, and ultraviolet spectrums into electrical signals.Examples of image sensors may include digital cameras, phone cameras,semiconductor Charge-Coupled Devices (CCDs), active pixel sensors inComplementary Metal-Oxide-Semiconductor (CMOS), or N-typemetal-oxide-semiconductor (NMOS, Live MOS). The electrical signals maybe used to generate image data. Consistent with the present disclosure,the image data may include pixel data streams, digital images, digitalvideo streams, data derived from captured images, and data that may beused to construct one or more 3D images, a sequence of 3D images, 3Dvideos, or a virtual 3D representation. The image data acquired by imagesensor 372 may be transmitted by wired or wireless transmission to anyprocessing device of system 200. For example, the image data may beprocessed in order to: detect objects, detect events, detect action,detect face, detect people, recognize a known person, or any otherinformation that may be used by system 200. Consistent with the presentdisclosure, processing device 360 may modify a presentation of virtualcontent based on image data received from image sensor 372.

Motion sensor 373 may include one or more motion sensors configured tomeasure motion of input unit 202 or motion of objects in the environmentof input unit 202. Specifically, the motion sensors may perform at leastone of the following: detect motion of objects in the environment ofinput unit 202, measure the velocity of objects in the environment ofinput unit 202, measure the acceleration of objects in the environmentof input unit 202, detect the motion of input unit 202, measure thevelocity of input unit 202, measure the acceleration of input unit 202,etc. In some embodiments, motion sensor 373 may include one or moreaccelerometers configured to detect changes in proper accelerationand/or to measure proper acceleration of input unit 202. In otherembodiments, motion sensor 373 may include one or more gyroscopesconfigured to detect changes in the orientation of input unit 202 and/orto measure information related to the orientation of input unit 202. Inother embodiments, motion sensor 373 may include one or more using imagesensors, LIDAR sensors, radar sensors, or proximity sensors. Forexample, by analyzing captured images the processing device maydetermine the motion of input unit 202, for example, using ego-motionalgorithms. In addition, the processing device may determine the motionof objects in the environment of input unit 202, for example, usingobject tracking algorithms. Consistent with the present disclosure,processing device 360 may modify a presentation of virtual content basedon the determined motion of input unit 202 or the determined motion ofobjects in the environment of input unit 202. For example, causing avirtual display to follow the movement of input unit 202.

Environmental sensor 374 may include one or more sensors from differenttypes configured to capture data reflective of the environment of inputunit 202. In some embodiments, environmental sensor 374 may include oneor more chemical sensors configured to perform at least one of thefollowing: measure chemical properties in the environment of input unit202, measure changes in the chemical properties in the environment ofinput unit 202, detect the present of chemicals in the environment ofinput unit 202, measure the concentration of chemicals in theenvironment of input unit 202. Examples of such chemical properties mayinclude: pH level, toxicity, and temperature. Examples of such chemicalsmay include: electrolytes, particular enzymes, particular hormones,particular proteins, smoke, carbon dioxide, carbon monoxide, oxygen,ozone, hydrogen, and hydrogen sulfide. In other embodiments,environmental sensor 374 may include one or more temperature sensorsconfigured to detect changes in the temperature of the environment ofinput unit 202 and/or to measure the temperature of the environment ofinput unit 202. In other embodiments, environmental sensor 374 mayinclude one or more barometers configured to detect changes in theatmospheric pressure in the environment of input unit 202 and/or tomeasure the atmospheric pressure in the environment of input unit 202.In other embodiments, environmental sensor 374 may include one or morelight sensors configured to detect changes in the ambient light in theenvironment of input unit 202. Consistent with the present disclosure,processing device 360 may modify a presentation of virtual content basedon input from environmental sensor 374. For example, automaticallyreducing the brightness of the virtual content when the environment ofuser 100 becomes darker.

Other sensors 375 may include a weight sensor, a light sensor, aresistive sensor, an ultrasonic sensor, a proximity sensor, a biometricsensor, or other sensing devices to facilitate related functionalities.In a specific embodiment, other sensors 375 may include one or morepositioning sensors configured to obtain positioning information ofinput unit 202, to detect changes in the position of input unit 202,and/or to measure the position of input unit 202. Alternatively, GPSsoftware may permit input unit 202 to access an external GPS receiver(e.g., connecting via a serial port or Bluetooth). Consistent with thepresent disclosure, processing device 360 may modify a presentation ofvirtual content based on input from other sensors 375. For example,presenting private information only after identifying user 100 usingdata from a biometric sensor.

The components and arrangements shown in FIG. 3 are not intended tolimit any embodiment. As will be appreciated by a person skilled in theart having the benefit of this disclosure, numerous variations and/ormodifications may be made to the depicted configuration of input unit202. For example, not all components may be essential for the operationof an input unit in all cases. Any component may be located in anyappropriate part of an input unit, and the components may be rearrangedinto a variety of configurations while providing the functionality ofvarious embodiments. For example, some input units may not include allof the elements as shown in input unit 202.

FIG. 4 is a block diagram of an examplary configuration of XR unit 204.FIG. 4 is an exemplary representation of just one embodiment, and it isto be understood that some illustrated elements might be omitted andothers added within the scope of this disclosure. In the embodiment ofFIG. 4, XR unit 204 may directly or indirectly access a bus 400 (orother communication mechanism) that interconnects subsystems andcomponents for transferring information within XR unit 204. For example,bus 400 may interconnect a memory interface 410, a network interface420, an input interface 430, a power source 440, an output interface450, a processing device 460, a sensors interface 470, and a database480.

Memory interface 410, shown in FIG. 4, is assumed to have similarfunctionality as the functionality of memory interface 310 describedabove in detail. Memory interface 410 may be used to access a softwareproduct and/or data stored on a non-transitory computer-readable mediumor on memory devices, such as memory device 411. Memory device 411 maycontain software modules to execute processes consistent with thepresent disclosure. In particular, memory device 411 may include aninput determination module 412, an output determination module 413, asensors communication module 414, a virtual content determination module415, a virtual content communication module 416, and a database accessmodule 417. Modules 412-417 may contain software instructions forexecution by at least one processor (e.g., processing device 460)associated with XR unit 204. Input determination module 412, outputdetermination module 413, sensors communication module 414, virtualcontent determination module 415, virtual content communication module416, and database access module 417 may cooperate to perform variousoperations. For example, input determination module 412 may determineUser Interface (UI) input received from input unit 202. At the sametime, sensors communication module 414 may receive data from differentsensors to determine a status of user 100. Virtual content determinationmodule 415 may determine the virtual content to display based onreceived input and the determined status of user 100. Virtual contentcommunication module 416 may retrieve virtual content not determined byvirtual content determination module 415. The retrieval of the virtualcontent may be from database 380, database 480, mobile communicationsdevice 206, or from remote processing unit 208. Based on the output ofvirtual content determination module 415, output determination module413 may cause a change in a virtual content displayed to user 100 byprojector 454.

In some embodiments, input determination module 412 may regulate theoperation of input interface 430 in order to receive gesture input 431,virtual input 432, audio input 433, and UI input 434. Consistent withthe present disclosure, input determination module 412 may concurrentlyreceive different types of input data. In one embodiment, inputdetermination module 412 may apply different rules based on the detectedtype of input. For example, gesture input may have precedence overvirtual input. In some embodiments, output determination module 413 mayregulate the operation of output interface 450 in order to generateoutput using light indicators 451, display 452, speakers 453, andprojector 454. In one embodiment, light indicators 451 may include alight indicator that shows the status of the wearable extended realityappliance. For example, the light indicator may display green light whenthe wearable extended reality appliance 110 are connected to input unit202, and blinks when wearable extended reality appliance 110 has lowbattery. In another embodiment, display 452 may be used to displayoperational information. In another embodiment, speakers 453 may includea bone conduction headphone used to output audio to user 100. In anotherembodiment, projector 454 may present virtual content to user 100.

The operations of a sensors communication module, a virtual contentdetermination module, a virtual content communication module, and adatabase access module are described above with reference to FIG. 3,details of which are not repeated herein. Modules 412-417 may beimplemented in software, hardware, firmware, a mix of any of those, orthe like.

Network interface 420, shown in FIG. 4, is assumed to have similarfunctionality as the functionality of network interface 320, describedabove in detail. The specific design and implementation of networkinterface 420 may depend on the communications network(s) over which XRunit 204 is intended to operate. For example, in some embodiments, XRunit 204 is configured to be selectively connectable by wire to inputunit 202. When connected by wire, network interface 420 may enablecommunications with input unit 202; and when not connected by wire,network interface 420 may enable communications with mobilecommunications device 206.

Input interface 430, shown in FIG. 4, is assumed to have similarfunctionality as the functionality of input interface 330 describedabove in detail. In this case, input interface 430 may communicate withan image sensor to obtain gesture input 431 (e.g., a finger of user 100pointing to a virtual object), communicate with other XR units 204 toobtain virtual input 432 (e.g., a virtual object shared with XR unit 204or a gesture of avatar detected in the virtual environment), communicatewith a microphone to obtain audio input 433 (e.g., voice commands), andcommunicate with input unit 202 to obtain UI input 434 (e.g., virtualcontent determined by virtual content determination module 315).

Power source 440, shown in FIG. 4, is assumed to have similarfunctionality as the functionality of power source 340 described above,only it provides electrical energy to power XR unit 204. In someembodiments, power source 440 may be charged by power source 340. Forexample, power source 440 may be wirelessly changed when XR unit 204 isplaced on or in proximity to input unit 202.

Output interface 450, shown in FIG. 4, is assumed to have similarfunctionality as the functionality of output interface 350 describedabove in detail. In this case, output interface 450 may cause outputfrom light indicators 451, display 452, speakers 453, and projector 454.Projector 454 may be any device, apparatus, instrument, or the likecapable of projecting (or directing) light in order to display virtualcontent onto a surface. The surface may be part of XR unit 204, part ofan eye of user 100, or part of an object in proximity to user 100. Inone embodiment, projector 454 may include a lighting unit thatconcentrates light within a limited solid angle by means of one or moremirrors and lenses, and provides a high value of luminous intensity in adefined direction.

Processing device 460, shown in FIG. 4, is assumed to have similarfunctionality as the functionality of processing device 360 describedabove in detail. When XR unit 204 is connected to input unit 202,processing device 460 may work together with processing device 360.Specifically, processing device 460 may implement virtual machinetechnologies or other technologies to provide the ability to execute,control, run, manipulate, store, etc., multiple software processes,applications, programs, etc. It is appreciated that other types ofprocessor arrangements could be implemented to provide the capabilitiesdisclosed herein.

Sensors interface 470, shown in FIG. 4, is assumed to have similarfunctionality as the functionality of sensors interface 370 describedabove in detail. Specifically sensors interface 470 may communicate withaudio sensor 471, image sensor 472, motion sensor 473, environmentalsensor 474, and other sensors 475. The operations of an audio sensor, animage sensor, a motion sensor, an environmental sensor, and othersensors are described above with reference to FIG. 3, details of whichare not repeated herein. It is appreciated that other types andcombination of sensors may be used to provide the capabilities disclosedherein.

The components and arrangements shown in FIG. 4 are not intended tolimit any embodiment. As will be appreciated by a person skilled in theart having the benefit of this disclosure, numerous variations and/ormodifications may be made to the depicted configuration of XR unit 204.For example, not all components may be essential for the operation of XRunit 204 in all cases. Any component may be located in any appropriatepart of system 200, and the components may be rearranged into a varietyof configurations while providing the functionality of variousembodiments. For example, some XR units may not include all of theelements in XR unit 204 (e.g., wearable extended reality appliance 110may not have light indicators 451).

FIG. 5 is a block diagram of an examplary configuration of remoteprocessing unit 208. FIG. 5 is an exemplary representation of just oneembodiment, and it is to be understood that some illustrated elementsmight be omitted, and others added within the scope of this disclosure.In the embodiment of FIG. 5, remote processing unit 208 may include aserver 210 that directly or indirectly accesses a bus 500 (or othercommunication mechanism) interconnecting subsystems and components fortransferring information within server 210. For example, bus 500 mayinterconnect a memory interface 510, a network interface 520, a powersource 540, a processing device 560, and a database 580. Remoteprocessing unit 208 may also include a one or more data structures. Forexample, data structures 212A, 212B, and 212C.

Memory interface 510, shown in FIG. 5, is assumed to have similarfunctionality as the functionality of memory interface 310 describedabove in detail. Memory interface 510 may be used to access a softwareproduct and/or data stored on a non-transitory computer-readable mediumor on other memory devices, such as memory devices 311, 411, 511, ordata structures 212A, 212B, and 212C. Memory device 511 may containsoftware modules to execute processes consistent with the presentdisclosure. In particular, memory device 511 may include a shared memorymodule 512, a node registration module 513, a load balancing module 514,one or more computational nodes 515, an internal communication module516, an external communication module 517, and a database access module(not shown). Modules 512-517 may contain software instructions forexecution by at least one processor (e.g., processing device 560)associated with remote processing unit 208. Shared memory module 512,node registration module 513, load balancing module 514, computationalmodule 515, and external communication module 517 may cooperate toperform various operations.

Shared memory module 512 may allow information sharing between remoteprocessing unit 208 and other components of system 200. In someembodiments, shared memory module 512 may be configured to enableprocessing device 560 (and other processing devices in system 200) toaccess, retrieve, and store data. For example, using shared memorymodule 512, processing device 560 may perform at least one of: executingsoftware programs stored on memory device 511, database 580, or datastructures 212A-C; storing information in memory device 511, database580, or data structures 212A-C; or retrieving information from memorydevice 511, database 580, or data structures 212A-C.

Node registration module 513 may be configured to track the availabilityof one or more computational nodes 515. In some examples, noderegistration module 513 may be implemented as: a software program, suchas a software program executed by one or more computational nodes 515, ahardware solution, or a combined software and hardware solution. In someimplementations, node registration module 513 may communicate with oneor more computational nodes 515, for example, using internalcommunication module 516. In some examples, one or more computationalnodes 515 may notify node registration module 513 of their status, forexample, by sending messages: at startup, at shutdown, at constantintervals, at selected times, in response to queries received from noderegistration module 513, or at any other determined times. In someexamples, node registration module 513 may query about the status of oneor more computational nodes 515, for example, by sending messages: atstartup, at constant intervals, at selected times, or at any otherdetermined times.

Load balancing module 514 may be configured to divide the workload amongone or more computational nodes 515. In some examples, load balancingmodule 514 may be implemented as: a software program, such as a softwareprogram executed by one or more of the computational nodes 515, ahardware solution, or a combined software and hardware solution. In someimplementations, load balancing module 514 may interact with noderegistration module 513 in order to obtain information regarding theavailability of one or more computational nodes 515. In someimplementations, load balancing module 514 may communicate with one ormore computational nodes 515, for example, using internal communicationmodule 516. In some examples, one or more computational nodes 515 maynotify load balancing module 514 of their status, for example, bysending messages: at startup, at shutdown, at constant intervals, atselected times, in response to queries received from load balancingmodule 514, or at any other determined times. In some examples, loadbalancing module 514 may query about the status of one or morecomputational nodes 515, for example, by sending messages: at startup,at constant intervals, at pre-selected times, or at any other determinedtimes.

Internal communication module 516 may be configured to receive and/or totransmit information from one or more components of remote processingunit 208. For example, control signals and/or synchronization signalsmay be sent and/or received through internal communication module 516.In one embodiment, input information for computer programs, outputinformation of computer programs, and/or intermediate information ofcomputer programs may be sent and/or received through internalcommunication module 516. In another embodiment, information receivedthough internal communication module 516 may be stored in memory device511, in database 580, in data structures 212A-C, or other memory devicein system 200. For example, information retrieved from data structure212A may be transmitted using internal communication module 516. Inanother example, input data may be received using internal communicationmodule 516 and stored in data structure 212B.

External communication module 517 may be configured to receive and/or totransmit information from one or more components of system 200. Forexample, control signals may be sent and/or received through externalcommunication module 517. In one embodiment, information received thoughexternal communication module 517 may be stored in memory device 511, indatabase 580, in data structures 212A-C, and or any memory device in thesystem 200. In another embodiment, information retrieved from any ofdata structures 212A-C may be transmitted using external communicationmodule 517 to XR unit 204. In another embodiment, input data may betransmitted and/or received using external communication module 517.Examples of such input data may include data received from input unit202, information captured from the environment of user 100 using one ormore sensors (e.g., audio sensor 471, image sensor 472, motion sensor473, environmental sensor 474, other sensors 475), and more.

In some embodiments, aspects of modules 512-517 may be implemented inhardware, in software (including in one or more signal processing and/orapplication specific integrated circuits), in firmware, or in anycombination thereof, executable by one or more processors, alone, or invarious combinations with each other. Specifically, modules 512-517 maybe configured to interact with each other and/or other modules of system200 to perform functions consistent with embodiments of the presentdisclosure. Memory device 511 may include additional modules andinstructions or fewer modules and instructions.

Network interface 520, power source 540, processing device 560, anddatabase 580, shown in FIG. 5, are assumed to have similar functionalityas the functionality of similar elements described above with referenceto FIGS. 4 and 5. The specific design and implementation of theabove-mentioned components may vary based on the implementation ofsystem 200. In addition, remote processing unit 208 may include more orfewer components. For example, remote processing unit 208 may include aninput interface configured to receive direct input from one or moreinput devices.

Consistent with the present disclosure, a processing device of system200 (e.g., processor within mobile communications device 206, aprocessor within a server 210, a processor within a wearable extendedreality appliance, such as, wearable extended reality appliance 110,and/or a processor within an input device associated with wearableextended reality appliance 110, such as keyboard 104) may use machinelearning algorithms in order to implement any of the methods disclosedherein. In some embodiments, machine learning algorithms (also referredto as machine learning models in the present disclosure) may be trainedusing training examples, for example in the cases described below. Somenon-limiting examples of such machine learning algorithms may includeclassification algorithms, data regressions algorithms, imagesegmentation algorithms, visual detection algorithms (such as objectdetectors, face detectors, person detectors, motion detectors, edgedetectors, etc.), visual recognition algorithms (such as facerecognition, person recognition, object recognition, etc.), speechrecognition algorithms, mathematical embedding algorithms, naturallanguage processing algorithms, support vector machines, random forests,nearest neighbors algorithms, deep learning algorithms, artificialneural network algorithms, convolutional neural network algorithms,recurrent neural network algorithms, linear machine learning models,non-linear machine learning models, ensemble algorithms, and more. Forexample, a trained machine learning algorithm may comprise an inferencemodel, such as a predictive model, a classification model, a dataregression model, a clustering model, a segmentation model, anartificial neural network (such as a deep neural network, aconvolutional neural network, a recurrent neural network, etc.), arandom forest, a support vector machine, and so forth. In some examples,the training examples may include example inputs together with thedesired outputs corresponding to the example inputs. Further, in someexamples, training machine learning algorithms using the trainingexamples may generate a trained machine learning algorithm, and thetrained machine learning algorithm may be used to estimate outputs forinputs not included in the training examples. In some examples,engineers, scientists, processes and machines that train machinelearning algorithms may further use validation examples and/or testexamples. For example, validation examples and/or test examples mayinclude example inputs together with the desired outputs correspondingto the example inputs, a trained machine learning algorithm and/or anintermediately trained machine learning algorithm may be used toestimate outputs for the example inputs of the validation examplesand/or test examples, the estimated outputs may be compared to thecorresponding desired outputs, and the trained machine learningalgorithm and/or the intermediately trained machine learning algorithmmay be evaluated based on a result of the comparison. In some examples,a machine learning algorithm may have parameters and hyper parameters,where the hyper parameters may be set manually by a person orautomatically by a process external to the machine learning algorithm(such as a hyper parameter search algorithm), and the parameters of themachine learning algorithm may be set by the machine learning algorithmbased on the training examples. In some implementations, thehyper-parameters may be set based on the training examples and thevalidation examples, and the parameters may be set based on the trainingexamples and the selected hyper-parameters. For example, given thehyper-parameters, the parameters may be conditionally independent of thevalidation examples.

In some embodiments, trained machine learning algorithms (also referredto as machine learning models and trained machine learning models in thepresent disclosure) may be used to analyze inputs and generate outputs,for example in the cases described below. In some examples, a trainedmachine learning algorithm may be used as an inference model that whenprovided with an input generates an inferred output. For example, atrained machine learning algorithm may include a classificationalgorithm, the input may include a sample, and the inferred output mayinclude a classification of the sample (such as an inferred label, aninferred tag, and so forth). In another example, a trained machinelearning algorithm may include a regression model, the input may includea sample, and the inferred output may include an inferred valuecorresponding to the sample. In yet another example, a trained machinelearning algorithm may include a clustering model, the input may includea sample, and the inferred output may include an assignment of thesample to at least one cluster. In an additional example, a trainedmachine learning algorithm may include a classification algorithm, theinput may include an image, and the inferred output may include aclassification of an item depicted in the image. In yet another example,a trained machine learning algorithm may include a regression model, theinput may include an image, and the inferred output may include aninferred value corresponding to an item depicted in the image (such asan estimated property of the item, such as size, volume, age of a persondepicted in the image, distance from an item depicted in the image, andso forth). In an additional example, a trained machine learningalgorithm may include an image segmentation model, the input may includean image, and the inferred output may include a segmentation of theimage. In yet another example, a trained machine learning algorithm mayinclude an object detector, the input may include an image, and theinferred output may include one or more detected objects in the imageand/or one or more locations of objects within the image. In someexamples, the trained machine learning algorithm may include one or moreformulas and/or one or more functions and/or one or more rules and/orone or more procedures, the input may be used as input to the formulasand/or functions and/or rules and/or procedures, and the inferred outputmay be based on the outputs of the formulas and/or functions and/orrules and/or procedures (for example, selecting one of the outputs ofthe formulas and/or functions and/or rules and/or procedures, using astatistical measure of the outputs of the formulas and/or functionsand/or rules and/or procedures, and so forth).

Consistent with the present disclosure, a processing device of system200 may analyze image data captured by an image sensor (e.g., imagesensor 372, image sensor 472, or any other image sensor) in order toimplement any of the methods disclosed herein. In some embodiments,analyzing the image data may comprise analyzing the image data to obtaina preprocessed image data, and subsequently analyzing the image dataand/or the preprocessed image data to obtain the desired outcome. One ofordinary skill in the art will recognize that the followings areexamples, and that the image data may be preprocessed using other kindsof preprocessing methods. In some examples, the image data may bepreprocessed by transforming the image data using a transformationfunction to obtain a transformed image data, and the preprocessed imagedata may comprise the transformed image data. For example, thetransformed image data may comprise one or more convolutions of theimage data. For example, the transformation function may comprise one ormore image filters, such as low-pass filters, high-pass filters,band-pass filters, all-pass filters, and so forth. In some examples, thetransformation function may comprise a nonlinear function. In someexamples, the image data may be preprocessed by smoothing at least partsof the image data, for example using Gaussian convolution, using amedian filter, and so forth. In some examples, the image data may bepreprocessed to obtain a different representation of the image data. Forexample, the preprocessed image data may comprise: a representation ofat least part of the image data in a frequency domain; a DiscreteFourier Transform of at least part of the image data; a Discrete WaveletTransform of at least part of the image data; a time/frequencyrepresentation of at least part of the image data; a representation ofat least part of the image data in a lower dimension; a lossyrepresentation of at least part of the image data; a losslessrepresentation of at least part of the image data; a time ordered seriesof any of the above; any combination of the above; and so forth. In someexamples, the image data may be preprocessed to extract edges, and thepreprocessed image data may comprise information based on and/or relatedto the extracted edges. In some examples, the image data may bepreprocessed to extract image features from the image data. Somenon-limiting examples of such image features may comprise informationbased on and/or related to: edges; corners; blobs; ridges; ScaleInvariant Feature Transform (SIFT) features; temporal features; and soforth. In some examples, analyzing the image data may includecalculating at least one convolution of at least a portion of the imagedata, and using the calculated at least one convolution to calculate atleast one resulting value and/or to make determinations,identifications, recognitions, classifications, and so forth.

Consistent with another aspects of the disclosure, a processing deviceof system 200 may analyze image data in order to implement any of themethods disclosed herein. In some embodiments, analyzing the image maycomprise analyzing the image data and/or the preprocessed image datausing one or more rules, functions, procedures, artificial neuralnetworks, object detection algorithms, face detection algorithms, visualevent detection algorithms, action detection algorithms, motiondetection algorithms, background subtraction algorithms, inferencemodels, and so forth. Some non-limiting examples of such inferencemodels may include: an inference model preprogrammed manually; aclassification model; a regression model; a result of trainingalgorithms, such as machine learning algorithms and/or deep learningalgorithms, on training examples, where the training examples mayinclude examples of data instances, and in some cases, a data instancemay be labeled with a corresponding desired label and/or result, andmore. In some embodiments, analyzing image data (for example by themethods, steps and modules described herein) may comprise analyzingpixels, voxels, point cloud, range data, etc. included in the imagedata.

A convolution may include a convolution of any dimension. Aone-dimensional convolution is a function that transforms an originalsequence of numbers to a transformed sequence of numbers. Theone-dimensional convolution may be defined by a sequence of scalars.Each particular value in the transformed sequence of numbers may bedetermined by calculating a linear combination of values in asubsequence of the original sequence of numbers corresponding to theparticular value. A result value of a calculated convolution may includeany value in the transformed sequence of numbers. Likewise, ann-dimensional convolution is a function that transforms an originaln-dimensional array to a transformed array. The n-dimensionalconvolution may be defined by an n-dimensional array of scalars (knownas the kernel of the n-dimensional convolution). Each particular valuein the transformed array may be determined by calculating a linearcombination of values in an n-dimensional region of the original arraycorresponding to the particular value. A result value of a calculatedconvolution may include any value in the transformed array. In someexamples, an image may comprise one or more components (such as colorcomponents, depth component, etc.), and each component may include atwo-dimensional array of pixel values. In one example, calculating aconvolution of an image may include calculating a two-dimensionalconvolution on one or more components of the image. In another example,calculating a convolution of an image may include stacking arrays fromdifferent components to create a three-dimensional array, andcalculating a three-dimensional convolution on the resultingthree-dimensional array. In some examples, a video may comprise one ormore components (such as color components, depth component, etc.), andeach component may include a three-dimensional array of pixel values(with two spatial axes and one temporal axis). In one example,calculating a convolution of a video may include calculating athree-dimensional convolution on one or more components of the video. Inanother example, calculating a convolution of a video may includestacking arrays from different components to create a four-dimensionalarray, and calculating a four-dimensional convolution on the resultingfour dimensional array.

The following detailed description refers to the accompanying drawings.Wherever possible, the same reference numbers are used in the drawingsand the following description to refer to the same or similar parts.While several illustrative embodiments are described herein,modifications, adaptations and other implementations are possible. Forexample, substitutions, additions, or modifications may be made to thecomponents illustrated in the drawings, and the illustrative methodsdescribed herein may be modified by substituting, reordering, removing,or adding steps to the disclosed methods. Accordingly, the followingdetailed description is not limited to the disclosed embodiments andexamples, but is inclusive of general principles described herein andillustrated in the figures in addition to the general principlesencompassed by the appended claims.

The present disclosure is directed to systems and methods for providingusers an extended reality environment. The term “extended realityenvironment,” which may also be referred to as “extended reality,”“extended reality space,” or “extended environment,” refers to all typesof real- and-virtual combined environments and human-machineinteractions at least partially generated by computer technology. Theextended reality environment may be a completely simulated virtualenvironment or a combined real- and-virtual environment that a user mayperceive from different perspectives. In some examples, the user mayinteract with elements of the extended reality environment. Onenon-limiting example of an extended reality environment may be a virtualreality environment, also known as “virtual reality” or a “virtualenvironment.” An immersive virtual reality environment may be asimulated non-physical environment which provides to the user theperception of being present in the virtual environment. Anothernon-limiting example of an extended reality environment may be anaugmented reality environment, also known as “augmented reality” or“augmented environment.” An augmented reality environment may involvelive direct or indirect view of a physical real-world environment thatis enhanced with virtual computer-generated perceptual information, suchas virtual objects that the user may interact with. Another non-limitingexample of an extended reality environment is a mixed realityenvironment, also known as “mixed reality” or a “mixed environment.” Amixed reality environment may be a hybrid of physical real-world andvirtual environments, in which physical and virtual objects may coexistand interact in real time. In some examples, both augmented realityenvironments and mixed reality environments may include a combination ofreal and virtual worlds, real-time interactions, and accurate 3Dregistration of virtual and real objects. In some examples, bothaugmented reality environment and mixed reality environments may includeconstructive overlaid sensory information that may be added to thephysical environment. In other examples, both augmented realityenvironment and mixed reality environments may include destructivevirtual content that may mask at least part of the physical environment.

In some embodiments, the systems and methods may provide the extendedreality environment using an extended reality appliance. The termextended reality appliance may include any type of device or system thatenables a user to perceive and/or interact with an extended realityenvironment. The extended reality appliance may enable the user toperceive and/or interact with an extended reality environment throughone or more sensory modalities. Some non-limiting examples of suchsensory modalities may include visual, auditory, haptic, somatosensory,and olfactory. One example of the extended reality appliance is avirtual reality appliance that enables the user to perceive and/orinteract with a virtual reality environment. Another example of theextended reality appliance is an augmented reality appliance thatenables the user to perceive and/or interact with an augmented realityenvironment. Yet another example of the extended reality appliance is amixed reality appliance that enables the user to perceive and/orinteract with a mixed reality environment.

Consistent with one aspect of the disclosure, the extended realityappliance may be a wearable device, such as a head-mounted device, forexample, smart glasses, smart contact lens, headsets or any other deviceworn by a human for purposes of presenting an extended reality to thehuman. Other extended reality appliances may include holographicprojector or any other device or system capable of providing anaugmented reality (AR), virtual reality (VR), mixed reality (MR), or anyimmersive experience. Typical components of wearable extended realityappliances may include at least one of: a stereoscopic head-mounteddisplay, a stereoscopic head-mounted sound system, head-motion trackingsensors (such as gyroscopes, accelerometers, magnetometers, imagesensors, structured light sensors, etc.), head mounted projectors,eye-tracking sensors, and additional components described below.Consistent with another aspect of the disclosure, the extended realityappliance may be a non-wearable extended reality appliance.Specifically, the non-wearable extended reality appliance may includemulti-projected environment appliances. In some embodiments, an extendedreality appliance may be configured to change the viewing perspective ofthe extended reality environment in response to movements of the userand in response to head movements of the user in particular. In oneexample, a wearable extended reality appliance may change thefield-of-view of the extended reality environment in response to achange of the head pose of the user, such as by changing the spatialorientation without changing the spatial position of the user in theextended reality environment. In another example, a non-wearableextended reality appliance may change the spatial position of the userin the extended reality environment in response to a change in theposition of the user in the real world, for example, by changing thespatial position of the user in the extended reality environment withoutchanging the direction of the field-of-view with respect to the spatialposition.

According to some embodiments, an extended reality appliance may includea digital communication device configured to at least one of: receivingvirtual content data configured to enable a presentation of the virtualcontent, transmitting virtual content for sharing with at least oneexternal device, receiving contextual data from at least one externaldevice, transmitting contextual data to at least one external device,transmitting of usage data indicative of usage of the extended realityappliance, and transmitting of data based on information captured usingat least one sensor included in the extended reality appliance. Inadditional embodiments, the extended reality appliance may includememory for storing at least one of virtual data configured to enable apresentation of virtual content, contextual data, usage data indicativeof usage of the extended reality appliance, sensor data based oninformation captured using at least one sensor included in the wearableextended reality appliance, software instructions configured to cause aprocessing device to present the virtual content, software instructionsconfigured to cause a processing device to collect and analyze thecontextual data, software instructions configured to cause a processingdevice to collect and analyze the usage data, and software instructionsconfigured to cause a processing device to collect and analyze thesensor data. In additional embodiments, the extended reality appliancemay include a processing device configured to perform at least one ofrendering of virtual content, collecting and analyzing contextual data,collecting and analyzing usage data, and collecting and analyzing sensordata. In additional embodiments, the extended reality appliance mayinclude one or more sensors. The one or more sensors may include one ormore image sensors (e.g., configured to capture images and/or videos ofa user of the appliance or of an environment of the user), one or moremotion sensors (such as an accelerometer, a gyroscope, a magnetometer,etc.), one or more positioning sensors (such as GPS, outdoor positioningsensor, indoor positioning sensor, etc.), one or more temperaturesensors (e.g., configured to measure the temperature of at least part ofthe appliance and/or of the environment), one or more contact sensors,one or more proximity sensors (e.g., configured to detect whether theappliance is currently worn), one or more electrical impedance sensors(e.g., configured to measure electrical impedance of the user), one ormore eye tracking sensors, such as gaze detectors, optical trackers,electric potential trackers (e.g., electrooculogram (EOG) sensors),video-based eye-trackers, infra-red/near infra-red sensors, passivelight sensors, or any other technology capable of determining where ahuman is looking or gazing.

In some embodiments, the systems and methods may use an input device tointeract with the extended reality appliance. The term input device mayinclude any physical device configured to receive input from a user oran environment of the user, and to provide the data to a computationaldevice. The data provided to the computational device may be in adigital format and/or in an analog format. In one embodiment, the inputdevice may store the input received from the user in a memory deviceaccessible by a processing device, and the processing device may accessthe stored data for analysis. In another embodiment, the input devicemay provide the data directly to a processing device, for example, overa bus or over another communication system configured to transfer datafrom the input device to the processing device. In some examples, theinput received by the input device may include key presses, tactileinput data, motion data, position data, gestures based input data,direction data, or any other data for supply for computation. Someexamples of the input device may include a button, a key, a keyboard, acomputer mouse, a touchpad, a touchscreen, a joystick, or anothermechanism from which input may be received. Another example of an inputdevice may include an integrated computational interface device thatincludes at least one physical component for receiving input from auser. The integrated computational interface device may include at leasta memory, a processing device, and the at least one physical componentfor receiving input from a user. In one example, the integratedcomputational interface device may further include a digital networkinterface that enables digital communication with other computingdevices. In one example, the integrated computational interface devicemay further include a physical component for outputting information tothe user. In some examples, all components of the integratedcomputational interface device may be included in a single housing,while in other examples the components may be distributed among two ormore housings. Some non-limiting examples of physical components forreceiving input from users that may be included in the integratedcomputational interface device may include at least one of a button, akey, a keyboard, a touchpad, a touchscreen, a joystick, or any othermechanism or sensor from which computational information may bereceived. Some non-limiting examples of physical components foroutputting information to users may include at least one of a lightindicator (such as a LED indicator), a screen, a touchscreen, a beeper,an audio speaker, or any other audio, video, or haptic device thatprovides human-perceptible outputs.

In some embodiments, image data may be captured using one or more imagesensors. In some examples, the image sensors may be included in theextended reality appliance, in a wearable device, in the wearableextended reality device, in the input device, in an environment of auser, and so forth. In some examples, the image data may be read frommemory, may be received from an external device, may be generated (forexample, using a generative model), and so forth. Some non-limitingexamples of image data may include images, grayscale images, colorimages, 2D images, 3D images, videos, 2D videos, 3D videos, frames,footages, data derived from other image data, and so forth. In someexamples, the image data may be encoded in any analog or digital format.Some non-limiting examples of such formats may include raw formats,compressed formats, uncompressed formats, lossy formats, losslessformats, JPEG, GIF, PNG, TIFF, BMP, NTSC, PAL, SECAM, MPEG, MPEG-4 Part14, MOV, WMV, FLV, AVI, AVCHD, WebM, MKV, and so forth.

In some embodiments, the extended reality appliance may receive digitalsignals, for example, from the input device. The term digital signalsmay refer to a series of digital values that are discrete in time. Thedigital signals may represent, for example, sensor data, textual data,voice data, video data, virtual data, or any other form of data thatprovides perceptible information. Consistent with the presentdisclosure, the digital signals may be configured to cause the extendedreality appliance to present virtual content. In one embodiment, thevirtual content may be presented in a selected orientation. In thisembodiment, the digital signals may indicate a position and an angle ofa viewpoint in an environment, such as an extended reality environment.Specifically, the digital signals may include an encoding of theposition and angle in six degree-of-freedom coordinates (e.g.,forward/back, up/down, left/right, yaw, pitch, and roll). In anotherembodiment, the digital signals may include an encoding of the positionas three-dimensional coordinates (e.g., x, y, and z), and an encoding ofthe angle as a vector originating from the encoded position.Specifically, the digital signals may indicate the orientation and anangle of the presented virtual content in an absolute coordinates of theenvironment, for example, by encoding yaw, pitch and roll of the virtualcontent with respect to a standard default angle. In another embodiment,the digital signals may indicate the orientation and the angle of thepresented virtual content with respect to a viewpoint of another object(e.g., a virtual object, a physical object, etc.), for example, byencoding yaw, pitch, and roll of the virtual content with respect adirection corresponding to the viewpoint or to a direction correspondingto the other object. In another embodiment, such digital signals mayinclude one or more projections of the virtual content, for example, ina format ready for presentation (e.g., image, video, etc.). For example,each such projection may correspond to a particular orientation or aparticular angle. In another embodiment, the digital signals may includea representation of virtual content, for example, by encoding objects ina three-dimensional array of voxels, in a polygon mesh, or in any otherformat in which virtual content may be presented.

In some embodiments, the digital signals may be configured to cause theextended reality appliance to present virtual content. The term virtualcontent may include any type of data representation that may bedisplayed by the extended reality appliance to the user. The virtualcontent may include a virtual object, inanimate virtual content, animatevirtual content configured to change over time or in response totriggers, virtual two-dimensional content, virtual three dimensionalcontent, a virtual overlay over a portion of a physical environment orover a physical object, a virtual addition to a physical environment orto a physical object, a virtual promotion content, a virtualrepresentation of a physical object, a virtual representation of aphysical environment, a virtual document, a virtual character orpersona, a virtual computer screen, a virtual widget, or any otherformat for displaying information virtually. Consistent with the presentdisclosure, the virtual content may include any visual presentationrendered by a computer or a processing device. In one embodiment, thevirtual content may include a virtual object that is a visualpresentation rendered by a computer in a confined region and configuredto represent an object of a particular type (such as an inanimatevirtual object, an animate virtual object, virtual furniture, a virtualdecorative object, virtual widget, or other virtual representation). Therendered visual presentation may change to reflect changes to a statusobject or changes in the viewing angle of the object, for example, in away that mimics changes in the appearance of physical objects. Inanother embodiment, the virtual content may include a virtual display(also referred to as a “virtual display screen” or a “virtual screen”herein), such as a virtual computer screen, a virtual tablet screen or avirtual smartphone screen, configured to display information generatedby an operating system, in which the operating system may be configuredto receive textual data from a physical keyboard and/or a virtualkeyboard and to cause a display of the textual content in the virtualdisplay screen. In one example, illustrated in FIG. 1, the virtualcontent may include a virtual environment that includes a virtualcomputer screen and a plurality of virtual objects. In some examples, avirtual display may be a virtual object mimicking and/or extending thefunctionality of a physical display screen. For example, the virtualdisplay may be presented in an extended reality environment (such as amixed reality environment, an augmented reality environment, a virtualreality environment, etc.), using an extended reality appliance. In oneexample, a virtual display may present content produced by a regularoperating system that may be equally presented on a physical displayscreen. In one example, a textual content entered using a keyboard (forexample, using a physical keyboard, using a virtual keyboard, etc.) maybe presented on a virtual display in real time as the textual content istyped. In one example, a virtual cursor may be presented on a virtualdisplay, and the virtual cursor may be controlled by a pointing device(such as a physical pointing device, a virtual pointing device, acomputer mouse, a joystick, a touchpad, a physical touch controller, andso forth). In one example, one or more windows of a graphical userinterface operating system may be presented on a virtual display. Inanother example, content presented on a virtual display may beinteractive, that is, it may change in reaction to actions of users. Inyet another example, a presentation of a virtual display may include apresentation of a screen frame, or may include no presentation of ascreen frame.

Some disclosed embodiments may include and/or access a data structure ora database. The terms data structure and a database, consistent with thepresent disclosure may include any collection of data values andrelationships among them. The data may be stored linearly, horizontally,hierarchically, relationally, non-relationally, uni-dimensionally,multidimensionally, operationally, in an ordered manner, in an unorderedmanner, in an object-oriented manner, in a centralized manner, in adecentralized manner, in a distributed manner, in a custom manner, or inany manner enabling data access. By way of non-limiting examples, datastructures may include an array, an associative array, a linked list, abinary tree, a balanced tree, a heap, a stack, a queue, a set, a hashtable, a record, a tagged union, Entity-Relationship model, a graph, ahypergraph, a matrix, a tensor, and so forth. For example, a datastructure may include an XML database, an RDBMS database, an SQLdatabase or NoSQL alternatives for data storage/search such as, forexample, MongoDB, Redis, Couchbase, Datastax Enterprise Graph, ElasticSearch, Splunk, Solr, Cassandra, Amazon DynamoDB, Scylla, HBase, andNeo4J. A data structure may be a component of the disclosed system or aremote computing component (e.g., a cloud-based data structure). Data inthe data structure may be stored in contiguous or non-contiguous memory.Moreover, a data structure, does not require information to beco-located. It may be distributed across multiple servers, for example,that may be owned or operated by the same or different entities. Thus,the term data structure in the singular is inclusive of plural datastructures.

In some embodiments, the system may determine the confidence level inreceived input or in any determined value. The term confidence levelrefers to any indication, numeric or otherwise, of a level (e.g., withina predetermined range) indicative of an amount of confidence the systemhas at determined data. For example, the confidence level may have avalue between 1 and 10. Alternatively, the confidence level may beexpressed as a percentage or any other numerical or non-numericalindication. In some cases, the system may compare the confidence levelto a threshold. The term threshold may denote a reference value, alevel, a point, or a range of values. In operation, when the confidencelevel of determined data exceeds the threshold (or is below it,depending on a particular use case), the system may follow a firstcourse of action and, when the confidence level is below it (or aboveit, depending on a particular use case), the system may follow a secondcourse of action. The value of the threshold may be predetermined foreach type of examined object or may be dynamically selected based ondifferent considerations.

Reference is now made to FIG. 1, which illustrates a user that uses anexample extended reality system consistent with embodiments of thepresent disclosure FIG. 1 is an exemplary representation of just oneembodiment, and it is to be understood that some illustrated elementsmight be omitted and others added within the scope of this disclosure.As shown, a user 100 is sitting behind table 102, supporting a keyboard104 and mouse 106. Keyboard 104 is connected by wire 108 to a wearableextended reality appliance 110 that displays virtual content to user100. Alternatively or additionally to wire 108, keyboard 104 may connectto wearable extended reality appliance 110 wirelessly. For illustrationpurposes, the wearable extended reality appliance is depicted a pair ofsmart glasses, but, as described above, wearable extended realityappliance 110 may be any type of head-mounted device used for presentingan extended reality to user 100. The virtual content displayed bywearable extended reality appliance 110 includes a virtual screen 112(also referred to as a “virtual display screen” or a “virtual display”herein) and a plurality of virtual widgets 114. Virtual widgets114A-114D are displayed next to virtual screen 112 and virtual widget114E is displayed on table 102. User 100 may input text to a document116 displayed in virtual screen 112 using keyboard 104; and may controlvirtual cursor 118 using mouse 106. In one example, virtual cursor 118may move anywhere within virtual screen 112. In another example, virtualcursor 118 may move anywhere within virtual screen 112 and may also moveto any one of virtual widgets 114A-114D but not to virtual widget 114E.In yet another example, virtual cursor 118 may move anywhere withinvirtual screen 112 and may also move to any one of virtual widgets114A-114E. In an additional example, virtual cursor 118 may moveanywhere in the extended reality environment including virtual screen112 and virtual widgets 114A-114E. In yet another example, virtualcursor may move on all available surfaces (i.e., virtual surfaces orphysical surfaces) or only on selected surfaces in the extended realityenvironment. Alternatively or additionally, user 100 may interact withany one of virtual widgets 114A-114E, or with selected virtual widgets,using hand gestures recognized by wearable extended reality appliance110. For example, virtual widget 114E may be an interactive widget(e.g., a virtual slider controller) that may be operated with handgestures.

FIG. 2 illustrates an example of a system 200 that provides extendedreality (XR) experience to users, such as user 100. FIG. 2 is anexemplary representation of just one embodiment, and it is to beunderstood that some illustrated elements might be omitted and othersadded within the scope of this disclosure. System 200 may becomputer-based and may include computer system components, wearableappliances, workstations, tablets, handheld computing devices, memorydevices, and/or internal network(s) connecting the components. System200 may include or be connected to various network computing resources(e.g., servers, routers, switches, network connections, storage devices,etc.) for supporting services provided by system 200. Consistent withthe present disclosure, system 200 may include an input unit 202, an XRunit 204, a mobile communications device 206, and a remote processingunit 208. Remote processing unit 208 may include a server 210 coupled toone or more physical or virtual storage devices, such as a datastructure 212. System 200 may also include or be connected to acommunications network 214 that facilitates communications and dataexchange between different system components and the different entitiesassociated with system 200.

Consistent with the present disclosure, input unit 202 may include oneor more devices that may receive input from user 100. In one embodiment,input unit 202 may include a textual input device, such as keyboard 104.The textual input device may include all possible types of devices andmechanisms for inputting textual information to system 200. Examples oftextual input devices may include mechanical keyboards, membranekeyboards, flexible keyboards, QWERTY keyboards, Dvorak keyboards,Colemak keyboards, chorded keyboards, wireless keyboards, keypads,key-based control panels, or other arrays of control keys, vision inputdevices, or any other mechanism for inputting text, whether themechanism is provided in physical form or is presented virtually. In oneembodiment, input unit 202 may also include a pointing input device,such as mouse 106. The pointing input device may include all possibletypes of devices and mechanisms for inputting two-dimensional orthree-dimensional information to system 200. In one example,two-dimensional input from the pointing input device may be used forinteracting with virtual content presented via the XR unit 204. Examplesof pointing input devices may include a computer mouse, trackball,touchpad, trackpad, touchscreen, joystick, pointing stick, stylus, lightpen, or any other physical or virtual input mechanism. In oneembodiment, input unit 202 may also include a graphical input device,such as a touchscreen configured to detect contact, movement, or breakof movement. The graphical input device may use any of a plurality oftouch sensitivity technologies, including, but not limited to,capacitive, resistive, infrared, and surface acoustic wave technologiesas well as other proximity sensor arrays or other elements fordetermining one or more points of contact. In one embodiment, input unit202 may also include one or more voice input devices, such as amicrophone. The voice input device may include all possible types ofdevices and mechanisms for inputting voice data to facilitatevoice-enabled functions, such as voice recognition, voice replication,digital recording, and telephony functions. In one embodiment, inputunit 202 may also include one or more image input devices, such as animage sensor, configured to capture image data. In one embodiment, inputunit 202 may also include one or more haptic gloves configured tocapture hands motion and pose data. In one embodiment, input unit 202may also include one or more proximity sensors configured to detectpresence and/or movement of objects in a selected region near thesensors.

In accordance with some embodiments, the system may include at least onesensor configured to detect and/or measure a property associated withthe user, the user's action, or user's environment. One example of theat least one sensor, is sensor 216 included in input unit 202. Sensor216 may be a motion sensor, a touch sensor, a light sensor, an infraredsensor, an audio sensor, an image sensor, a proximity sensor, apositioning sensor, a gyroscope, a temperature sensor, a biometricsensor, or any other sensing devices to facilitate relatedfunctionalities. Sensor 216 may be integrated with, or connected to, theinput devices or it may be separated from the input devices. In oneexample, a thermometer may be included in mouse 106 to determine thebody temperature of user 100. In another example, a positioning sensormay be integrated with keyboard 104 to determine movement of user 100relative to keyboard 104. Such positioning sensor may be implementedusing one of the following technologies: Global Positioning System(GPS), GLObal NAvigation Satellite System (GLONASS), Galileo globalnavigation system, BeiDou navigation system, other Global NavigationSatellite Systems (GNSS), Indian Regional Navigation Satellite System(IRNSS), Local Positioning Systems (LPS), Real-Time Location Systems(RTLS), Indoor Positioning System (IPS), Wi-Fi based positioningsystems, cellular triangulation, image based positioning technology,indoor positioning technology, outdoor positioning technology, or anyother positioning technology.

In accordance with some embodiments, the system may include one or moresensors for identifying a position and/or a movement of a physicaldevice (such as a physical input device, a physical computing device,keyboard 104, mouse 106, wearable extended reality appliance 110, and soforth). The one or more sensors may be included in the physical deviceor may be external to the physical device. In some examples, an imagesensor external to the physical device (for example, an image sensorincluded in another physical device) may be used to capture image dataof the physical device, and the image data may be analyzed to identifythe position and/or the movement of the physical device. For example,the image data may be analyzed using a visual object tracking algorithmto identify the movement of the physical device, may be analyzed using avisual object detection algorithm to identify the position of thephysical device (for example, relative to the image sensor, in a globalcoordinates system, etc.), and so forth. In some examples, an imagesensor included in the physical device may be used to capture imagedata, and the image data may be analyzed to identify the position and/orthe movement of the physical device. For example, the image data may beanalyzed using visual odometry algorithms to identify the position ofthe physical device, may be analyzed using an ego-motion algorithm toidentify movement of the physical device, and so forth. In someexamples, a positioning sensor, such as an indoor positioning sensor oran outdoor positioning sensor, may be included in the physical deviceand may be used to determine the position of the physical device. Insome examples, a motion sensor, such as an accelerometer or a gyroscope,may be included in the physical device and may be used to determine themotion of the physical device. In some examples, a physical device, suchas a keyboard or a mouse, may be configured to be positioned on aphysical surface. Such physical device may include an optical mousesensor (also known as non-mechanical tracking engine) aimed towards thephysical surface, and the output of the optical mouse sensor may beanalyzed to determine movement of the physical device with respect tothe physical surface.

Consistent with the present disclosure, XR unit 204 may include awearable extended reality appliance configured to present virtualcontent to user 100. One example of the wearable extended realityappliance is wearable extended reality appliance 110. Additionalexamples of wearable extended reality appliance may include a VirtualReality (VR) device, an Augmented Reality (AR) device, a Mixed Reality(MR) device, or any other device capable of generating extended realitycontent. Some non-limiting examples of such devices may include NrealLight, Magic Leap One, Varjo, Quest 1/2, Vive, and others. In someembodiments, XR unit 204 may present virtual content to user 100.Generally, an extended reality appliance may include all real-and-virtual combined environments and human-machine interactionsgenerated by computer technology and wearables. As mentioned above, theterm “extended reality” (XR) refers to a superset which includes theentire spectrum from “the complete real” to “the complete virtual.” Itincludes representative forms such as augmented reality (AR), mixedreality (MR), virtual reality (VR), and the areas interpolated amongthem. Accordingly, it is noted that the terms “XR appliance,” “ARappliance,” “VR appliance,” and “MR appliance” may be usedinterchangeably herein and may refer to any device of the variety ofappliances listed above.

Consistent with the present disclosure, the system may exchange datawith a variety of communication devices associated with users, forexample, mobile communications device 206. The term “communicationdevice” is intended to include all possible types of devices capable ofexchanging data using digital communications network, analogcommunication network or any other communications network configured toconvey data. In some examples, the communication device may include asmartphone, a tablet, a smartwatch, a personal digital assistant, adesktop computer, a laptop computer, an IoT device, a dedicatedterminal, a wearable communication device, and any other device thatenables data communications. In some cases, mobile communications device206 may supplement or replace input unit 202. Specifically, mobilecommunications device 206 may be associated with a physical touchcontroller that may function as a pointing input device. Moreover,mobile communications device 206 may also, for example, be used toimplement a virtual keyboard and replace the textual input device. Forexample, when user 100 steps away from table 102 and walks to the breakroom with his smart glasses, he may receive an email that requires aquick answer. In this case, the user may select to use his or her ownsmartwatch as the input device and to type the answer to the email whileit is virtually presented by the smart glasses.

Consistent with the present disclosure, embodiments of the system mayinvolve the usage of a cloud server. The term “cloud server” refers to acomputer platform that provides services via a network, such as theInternet. In the example embodiment illustrated in FIG. 2, server 210may use virtual machines that may not correspond to individual hardware.For example, computational and/or storage capabilities may beimplemented by allocating appropriate portions of desirablecomputation/storage power from a scalable repository, such as a datacenter or a distributed computing environment. Specifically, in oneembodiment, remote processing unit 208 may be used together with XR unit204 to provide the virtual content to user 100. In one exampleconfiguration, server 210 may be a cloud server that functions as theoperation system (OS) of the wearable extended reality appliance. In oneexample, server 210 may implement the methods described herein usingcustomized hard-wired logic, one or more Application Specific IntegratedCircuits (ASICs), Field Programmable Gate Arrays (FPGAs), firmware,and/or program logic which, in combination with the computer system,cause server 210 to be a special-purpose machine.

In some embodiments, server 210 may access data structure 212 todetermine, for example, virtual content to display user 100. Datastructure 212 may utilize a volatile or non-volatile, magnetic,semiconductor, tape, optical, removable, non-removable, other type ofstorage device or tangible or non-transitory computer-readable medium,or any medium or mechanism for storing information. Data structure 212may be part of server 210 or separate from server 210, as shown. Whendata structure 212 is not part of server 210, server 210 may exchangedata with data structure 212 via a communication link. Data structure212 may include one or more memory devices that store data andinstructions used to perform one or more features of the disclosedmethods. In one embodiment, data structure 212 may include any of aplurality of suitable data structures, ranging from small datastructures hosted on a workstation to large data structures distributedamong data centers. Data structure 212 may also include any combinationof one or more data structures controlled by memory controller devices(e.g., servers) or software.

Consistent with the present disclosure, communications network may beany type of network (including infrastructure) that supportscommunications, exchanges information, and/or facilitates the exchangeof information between the components of a system. For example,communications network 214 in system 200 may include, for example, atelephone network, an extranet, an intranet, the Internet, satellitecommunications, off-line communications, wireless communications,transponder communications, a Local Area Network (LAN), wireless network(e.g., a Wi-Fi/302.11 network), a Wide Area Network (WAN), a VirtualPrivate Network (VPN), digital communication network, analogcommunication network, or any other mechanism or combinations ofmechanism that enable data transmission.

The components and arrangements of system 200 shown in FIG. 2 areintended to be exemplary only and are not intended to limit thedisclosed embodiments, as the system components used to implement thedisclosed processes and features may vary.

FIG. 3 is a block diagram of an examplary configuration of input unit202. FIG. 3 is an exemplary representation of just one embodiment, andit is to be understood that some illustrated elements might be omittedand others added within the scope of this disclosure. In the embodimentof FIG. 3, input unit 202 may directly or indirectly access a bus 300(or other communication mechanism) that interconnects subsystems andcomponents for transferring information within input unit 202. Forexample, bus 300 may interconnect a memory interface 310, a networkinterface 320, an input interface 330, a power source 340, an outputinterface 350, a processing device 360, a sensors interface 370, and adatabase 380.

Memory interface 310, shown in FIG. 3, may be used to access a softwareproduct and/or data stored on a non-transitory computer-readable medium.Generally, a non-transitory computer-readable storage medium refers toany type of physical memory on which information or data readable by atleast one processor can be stored. Examples include Random Access Memory(RAM), Read-Only Memory (ROM), volatile memory, nonvolatile memory, harddrives, CD ROMs, DVDs, flash drives, disks, any other optical datastorage medium, any physical medium with patterns of holes, a PROM, anEPROM, a FLASH-EPROM or any other flash memory, NVRAM, a cache, aregister, any other memory chip or cartridge, and networked versions ofthe same. The terms “memory” and “computer-readable storage medium” mayrefer to multiple structures, such as a plurality of memories orcomputer-readable storage mediums located within an input unit or at aremote location. Additionally, one or more computer-readable storagemediums can be utilized in implementing a computer-implemented method.Accordingly, the term computer-readable storage medium should beunderstood to include tangible items and exclude carrier waves andtransient signals. In the specific embodiment illustrated in FIG. 3,memory interface 310 may be used to access a software product and/ordata stored on a memory device, such as memory device 311. Memory device311 may include high-speed random-access memory and/or non-volatilememory, such as one or more magnetic disk storage devices, one or moreoptical storage devices, and/or flash memory (e.g., NAND, NOR).Consistent with the present disclosure, the components of memory device311 may be distributed in more than units of system 200 and/or in morethan one memory device.

Memory device 311, shown in FIG. 3, may contain software modules toexecute processes consistent with the present disclosure. In particular,memory device 311 may include an input determination module 312, anoutput determination module 313, a sensors communication module 314, avirtual content determination module 315, a virtual contentcommunication module 316, and a database access module 317. Modules312-317 may contain software instructions for execution by at least oneprocessor (e.g., processing device 360) associated with input unit 202.Input determination module 312, output determination module 313, sensorscommunication module 314, virtual content determination module 315,virtual content communication module 316, and database access module 317may cooperate to perform various operations. For example, inputdetermination module 312 may determine text using data received from,for example, keyboard 104. Thereafter, output determination module 313may cause presentation of the recent inputted text, for example on adedicated display 352 physically or wirelessly coupled to keyboard 104.This way, when user 100 types, he can see a preview of the typed textwithout constantly moving his head up and down to look at virtual screen112. Sensors communication module 314 may receive data from differentsensors to determine a status of user 100. Thereafter, virtual contentdetermination module 315 may determine the virtual content to display,based on received input and the determined status of user 100. Forexample, the determined virtual content may be a virtual presentation ofthe recent inputted text on a virtual screen virtually located adjacentto keyboard 104. Virtual content communication module 316 may obtainvirtual content that is not determined by virtual content determinationmodule 315 (e.g., an avatar of another user). The retrieval of thevirtual content may be from database 380, from remote processing unit208, or any other source.

In some embodiments, input determination module 312 may regulate theoperation of input interface 330 in order to receive pointer input 331,textual input 332, audio input 333, and XR-related input 334. Details onthe pointer input, the textual input, and the audio input are describedabove. The term “XR-related input” may include any type of data that maycause a change in the virtual content displayed to user 100. In oneembodiment, XR-related input 334 may include image data of user 100, awearable extended reality appliance (e.g., detected hand gestures ofuser 100). In another embodiment, XR-related input 334 may includewireless communication indicating a presence of another user inproximity to user 100. Consistent with the present disclosure, inputdetermination module 312 may concurrently receive different types ofinput data. Thereafter, input determination module 312 may further applydifferent rules based on the detected type of input. For example, apointer input may have precedence over voice input.

In some embodiments, output determination module 313 may regulate theoperation of output interface 350 in order to generate output usinglight indicators 351, display 352, and/or speakers 353. In general, theoutput generated by output determination module 313 does not includevirtual content to be presented by a wearable extended realityappliance. Instead, the output generated by output determination module313 include various outputs that relates to the operation of input unit202 and/or the operation of XR unit 204. In one embodiment, lightindicators 351 may include a light indicator that shows the status of awearable extended reality appliance. For example, the light indicatormay display green light when wearable extended reality appliance 110 areconnected to keyboard 104, and blinks when wearable extended realityappliance 110 has low battery. In another embodiment, display 352 may beused to display operational information. For example, the display maypresent error messages when the wearable extended reality appliance isinoperable. In another embodiment, speakers 353 may be used to outputaudio, for example, when user 100 wishes to play some music for otherusers.

In some embodiments, sensors communication module 314 may regulate theoperation of sensors interface 370 in order to receive sensor data fromone or more sensors, integrated with, or connected to, an input device.The one or more sensors may include: audio sensor 371, image sensor 372,motion sensor 373, environmental sensor 374 (e.g., a temperature sensor,ambient light detectors, etc.), and other sensors 375. In oneembodiment, the data received from sensors communication module 314 maybe used to determine the physical orientation of the input device. Thephysical orientation of the input device may be indicative of a state ofthe user and may be determined based on combination of a tilt movement,a roll movement, and a lateral movement. Thereafter, the physicalorientation of the input device may be used by virtual contentdetermination module 315 to modify display parameters of the virtualcontent to match the state of the user (e.g., attention, sleepy, active,sitting, standing, leaning backwards, leaning forward, walking, moving,riding, etc.).

In some embodiments, virtual content determination module 315 maydetermine the virtual content to be displayed by the wearable extendedreality appliance. The virtual content may be determined based on datafrom input determination module 312, sensors communication module 314,and other sources (e.g., database 380). In some embodiments, determiningthe virtual content may include determining the distance, the size, andthe orientation of the virtual objects. The determination of theposition of the virtual objects may be determined based on the type ofthe virtual objects. Specifically, with regards to the exampleillustrated in FIG. 1, the virtual content determination module 315 maydetermine to place four virtual widgets 114A-114D on the sides ofvirtual screen 112 and to place virtual widget 114E on table 102 becausevirtual widget 114E is a virtual controller (e.g., volume bar). Thedetermination of the position of the virtual objects may further bedetermined based on user's preferences. For example, for left-handedusers, virtual content determination module 315 may determine placing avirtual volume bar left of keyboard 104; and for right-handed users,virtual content determination module 315 may determine placing thevirtual volume bar right of keyboard 104.

In some embodiments, virtual content communication module 316 mayregulate the operation of network interface 320 in order to obtain datafrom one or more sources to be presented as virtual content to user 100.The one or more sources may include other XR units 204, the user'smobile communications device 206, remote processing unit 208, publiclyavailable information, etc. In one embodiment, virtual contentcommunication module 316 may communicate with mobile communicationsdevice 206 in order to provide a virtual representation of mobilecommunications device 206. For example, the virtual representation mayenable user 100 to read messages and interact with applicationsinstalled on the mobile communications device 206. Virtual contentcommunication module 316 may also regulate the operation of networkinterface 320 in order to share virtual content with other users. In oneexample, virtual content communication module 316 may use data frominput determination module to identify a trigger (e.g., the trigger mayinclude a gesture of the user) and to transfer content from the virtualdisplay to a physical display (e.g., TV) or to a virtual display of adifferent user.

In some embodiments, database access module 317 may cooperate withdatabase 380 to retrieve stored data. The retrieved data may include,for example, privacy levels associated with different virtual objects,the relationship between virtual objects and physical objects, theuser's preferences, the user's past behavior, and more. As describedabove, virtual content determination module 315 may use the data storedin database 380 to determine the virtual content. Database 380 mayinclude separate databases, including, for example, a vector database,raster database, tile database, viewport database, and/or a user inputdatabase. The data stored in database 380 may be received from modules314-317 or other components of system 200. Moreover, the data stored indatabase 380 may be provided as input using data entry, data transfer,or data uploading.

Modules 312-317 may be implemented in software, hardware, firmware, amix of any of those, or the like. In some embodiments, any one or moreof modules 312-317 and data associated with database 380 may be storedin XR unit 204, mobile communications device 206, or remote processingunit 208. Processing devices of system 200 may be configured to executethe instructions of modules 312-317. In some embodiments, aspects ofmodules 312-317 may be implemented in hardware, in software (includingin one or more signal processing and/or application specific integratedcircuits), in firmware, or in any combination thereof, executable by oneor more processors, alone, or in various combinations with each other.Specifically, modules 312-317 may be configured to interact with eachother and/or other modules of system 200 to perform functions consistentwith disclosed embodiments. For example, input unit 202 may executeinstructions that include an image processing algorithm on data from XRunit 204 to determine head movement of user 100. Furthermore, eachfunctionality described throughout the specification, with regards toinput unit 202 or with regards to a component of input unit 202, maycorrespond to a set of instructions for performing said functionality.These instructions need not be implemented as separate softwareprograms, procedures, or modules. Memory device 311 may includeadditional modules and instructions or fewer modules and instructions.For example, memory device 311 may store an operating system, such asANDROID, iOS, UNIX, OSX, WINDOWS, DARWIN, RTXC, LINUX or an embeddedoperating system such as VXWorkS. The operating system can includeinstructions for handling basic system services and for performinghardware-dependent tasks.

Network interface 320, shown in FIG. 3, may provide two-way datacommunications to a network, such as communications network 214. In oneembodiment, network interface 320 may include an Integrated ServicesDigital Network (ISDN) card, cellular modem, satellite modem, or a modemto provide a data communication connection over the Internet. As anotherexample, network interface 320 may include a Wireless Local Area Network(WLAN) card. In another embodiment, network interface 320 may include anEthernet port connected to radio frequency receivers and transmittersand/or optical (e.g., infrared) receivers and transmitters. The specificdesign and implementation of network interface 320 may depend on thecommunications network or networks over which input unit 202 is intendedto operate. For example, in some embodiments, input unit 202 may includenetwork interface 320 designed to operate over a GSM network, a GPRSnetwork, an EDGE network, a Wi-Fi or WiMax network, and a Bluetoothnetwork. In any such implementation, network interface 320 may beconfigured to send and receive electrical, electromagnetic, or opticalsignals that carry digital data streams or digital signals representingvarious types of information.

Input interface 330, shown in FIG. 3, may receive input from a varietyof input devices, for example, a keyboard, a mouse, a touch pad, a touchscreen, one or more buttons, a joystick, a microphone, an image sensor,and any other device configured to detect physical or virtual input. Thereceived input may be in the form of at least one of: text, sounds,speech, hand gestures, body gestures, tactile information, and any othertype of physically or virtually input generated by the user. In thedepicted embodiment, input interface 330 may receive pointer input 331,textual input 332, audio input 333, and XR-related input 334. Inadditional embodiment, input interface 330 may be an integrated circuitthat may act as bridge between processing device 360 and any of theinput devices listed above.

Power source 340, shown in FIG. 3, may provide electrical energy topower input unit 202 and optionally also power XR unit 204. Generally, apower source included in the any device or system in the presentdisclosure may be any device that can repeatedly store, dispense, orconvey electric power, including, but not limited to, one or morebatteries (e.g., a lead-acid battery, a lithium-ion battery, anickel-metal hydride battery, a nickel-cadmium battery), one or morecapacitors, one or more connections to external power sources, one ormore power convertors, or any combination of them. With reference to theexample illustrated in FIG. 3, the power source may be mobile, whichmeans that input unit 202 can be easily carried by a hand (e.g., thetotal weight of power source 340 may be less than a pound). The mobilityof the power source enables user 100 to use input unit 202 in a varietyof situations. In other embodiments, power source 340 may be associatedwith a connection to an external power source (such as an electricalpower grid) that may be used to charge power source 340. In addition,power source 340 may be configured to charge one or more batteriesincluded in XR unit 204; for example, a pair of extended reality glasses(e.g., wearable extended reality appliance 110) may be charged (e.g.,wirelessly or not wirelessly) when they are placed on or in proximity tothe input unit 202.

Output interface 350, shown in FIG. 3, may cause output from a varietyof output devices, for example, using light indicators 351, display 352,and/or speakers 353. In one embodiment, output interface 350 may be anintegrated circuit that may act as bridge between processing device 360and at least one of the output devices listed above. Light indicators351 may include one or more light sources, for example, a LED arrayassociated with different colors. Display 352 may include a screen(e.g., LCD or dot-matrix screen) or a touch screen. Speakers 353 mayinclude audio headphones, a hearing aid type device, a speaker, a boneconduction headphone, interfaces that provide tactile cues, vibrotactilestimulators, and more.

Processing device 360, shown in FIG. 3, may include at least oneprocessor configured to execute computer programs, applications,methods, processes, or other software to perform embodiments describedin the present disclosure. Generally, a processing device included inthe any device or system in the present disclosure may include one ormore integrated circuits, microchips, microcontrollers, microprocessors,all or part of a central processing unit (CPU), graphics processing unit(GPU), digital signal processor (DSP), field programmable gate array(FPGA), or other circuits suitable for executing instructions orperforming logic operations. The processing device may include at leastone processor configured to perform functions of the disclosed methodssuch as a microprocessor manufactured by Intel™. The processing devicemay include a single core or multiple core processors executing parallelprocesses simultaneously. In one example, the processing device may be asingle core processor configured with virtual processing technologies.The processing device may implement virtual machine technologies orother technologies to provide the ability to execute, control, run,manipulate, store, etc., multiple software processes, applications,programs, etc. In another example, the processing device may include amultiple-core processor arrangement (e.g., dual, quad core, etc.)configured to provide parallel processing functionalities to allow adevice associated with the processing device to execute multipleprocesses simultaneously. It is appreciated that other types ofprocessor arrangements could be implemented to provide the capabilitiesdisclosed herein.

Sensors interface 370, shown in FIG. 3, may obtain sensor data from avariety of sensors, for example, audio sensor 371, image sensor 372,motion sensor 373, environmental sensor 374, and other sensors 375. Inone embodiment, sensors interface 370 may be an integrated circuit thatmay act as bridge between processing device 360 and at least one of thesensors listed above.

Audio sensor 371 may include one or more audio sensors configured tocapture audio by converting sounds to digital information. Some examplesof audio sensors may include: microphones, unidirectional microphones,bidirectional microphones, cardioid microphones, omnidirectionalmicrophones, onboard microphones, wired microphones, wirelessmicrophones, or any combination of the above. Consistent with thepresent disclosure, processing device 360 may modify a presentation ofvirtual content based on data received from audio sensor 371 (e.g.,voice commands).

Image sensor 372 may include one or more image sensors configured tocapture visual information by converting light to image data. Consistentwith the present disclosure, an image sensor may be included in the anydevice or system in the present disclosure and may be any device capableof detecting and converting optical signals in the near-infrared,infrared, visible, and ultraviolet spectrums into electrical signals.Examples of image sensors may include digital cameras, phone cameras,semiconductor Charge-Coupled Devices (CCDs), active pixel sensors inComplementary Metal-Oxide-Semiconductor (CMOS), or N-typemetal-oxide-semiconductor (NMOS, Live MOS). The electrical signals maybe used to generate image data. Consistent with the present disclosure,the image data may include pixel data streams, digital images, digitalvideo streams, data derived from captured images, and data that may beused to construct one or more 3D images, a sequence of 3D images, 3Dvideos, or a virtual 3D representation. The image data acquired by imagesensor 372 may be transmitted by wired or wireless transmission to anyprocessing device of system 200. For example, the image data may beprocessed in order to: detect objects, detect events, detect action,detect face, detect people, recognize a known person, or any otherinformation that may be used by system 200. Consistent with the presentdisclosure, processing device 360 may modify a presentation of virtualcontent based on image data received from image sensor 372.

Motion sensor 373 may include one or more motion sensors configured tomeasure motion of input unit 202 or motion of objects in the environmentof input unit 202. Specifically, the motion sensors may perform at leastone of the following: detect motion of objects in the environment ofinput unit 202, measure the velocity of objects in the environment ofinput unit 202, measure the acceleration of objects in the environmentof input unit 202, detect the motion of input unit 202, measure thevelocity of input unit 202, measure the acceleration of input unit 202,etc. In some embodiments, motion sensor 373 may include one or moreaccelerometers configured to detect changes in proper accelerationand/or to measure proper acceleration of input unit 202. In otherembodiments, motion sensor 373 may include one or more gyroscopesconfigured to detect changes in the orientation of input unit 202 and/orto measure information related to the orientation of input unit 202. Inother embodiments, motion sensor 373 may include one or more using imagesensors, LIDAR sensors, radar sensors, or proximity sensors. Forexample, by analyzing captured images the processing device maydetermine the motion of input unit 202, for example, using ego-motionalgorithms. In addition, the processing device may determine the motionof objects in the environment of input unit 202, for example, usingobject tracking algorithms. Consistent with the present disclosure,processing device 360 may modify a presentation of virtual content basedon the determined motion of input unit 202 or the determined motion ofobjects in the environment of input unit 202. For example, causing avirtual display to follow the movement of input unit 202.

Environmental sensor 374 may include one or more sensors from differenttypes configured to capture data reflective of the environment of inputunit 202. In some embodiments, environmental sensor 374 may include oneor more chemical sensors configured to perform at least one of thefollowing: measure chemical properties in the environment of input unit202, measure changes in the chemical properties in the environment ofinput unit 202, detect the present of chemicals in the environment ofinput unit 202, measure the concentration of chemicals in theenvironment of input unit 202. Examples of such chemical properties mayinclude: pH level, toxicity, and temperature. Examples of such chemicalsmay include: electrolytes, particular enzymes, particular hormones,particular proteins, smoke, carbon dioxide, carbon monoxide, oxygen,ozone, hydrogen, and hydrogen sulfide. In other embodiments,environmental sensor 374 may include one or more temperature sensorsconfigured to detect changes in the temperature of the environment ofinput unit 202 and/or to measure the temperature of the environment ofinput unit 202. In other embodiments, environmental sensor 374 mayinclude one or more barometers configured to detect changes in theatmospheric pressure in the environment of input unit 202 and/or tomeasure the atmospheric pressure in the environment of input unit 202.In other embodiments, environmental sensor 374 may include one or morelight sensors configured to detect changes in the ambient light in theenvironment of input unit 202. Consistent with the present disclosure,processing device 360 may modify a presentation of virtual content basedon input from environmental sensor 374. For example, automaticallyreducing the brightness of the virtual content when the environment ofuser 100 becomes darker.

Other sensors 375 may include a weight sensor, a light sensor, aresistive sensor, an ultrasonic sensor, a proximity sensor, a biometricsensor, or other sensing devices to facilitate related functionalities.In a specific embodiment, other sensors 375 may include one or morepositioning sensors configured to obtain positioning information ofinput unit 202, to detect changes in the position of input unit 202,and/or to measure the position of input unit 202. Alternatively, GPSsoftware may permit input unit 202 to access an external GPS receiver(e.g., connecting via a serial port or Bluetooth). Consistent with thepresent disclosure, processing device 360 may modify a presentation ofvirtual content based on input from other sensors 375. For example,presenting private information only after identifying user 100 usingdata from a biometric sensor.

The components and arrangements shown in FIG. 3 are not intended tolimit the disclosed embodiments. As will be appreciated by a personskilled in the art having the benefit of this disclosure, numerousvariations and/or modifications may be made to the depictedconfiguration of input unit 202. For example, not all components may beessential for the operation of an input unit in all cases. Any componentmay be located in any appropriate part of an input unit, and thecomponents may be rearranged into a variety of configurations whileproviding the functionality of the disclosed embodiments. For example,some input units may not include all of the elements as shown in inputunit 202.

FIG. 4 is a block diagram of an examplary configuration of XR unit 204.FIG. 4 is an exemplary representation of just one embodiment, and it isto be understood that some illustrated elements might be omitted andothers added within the scope of this disclosure. In the embodiment ofFIG. 4, XR unit 204 may directly or indirectly access a bus 400 (orother communication mechanism) that interconnects subsystems andcomponents for transferring information within XR unit 204. For example,bus 400 may interconnect a memory interface 410, a network interface420, an input interface 430, a power source 440, an output interface450, a processing device 460, a sensors interface 470, and a database480.

Memory interface 410, shown in FIG. 4, is assumed to have similarfunctionality as the functionality of memory interface 310 describedabove in detail. Memory interface 410 may be used to access a softwareproduct and/or data stored on a non-transitory computer-readable mediumor on memory devices, such as memory device 411. Memory device 411 maycontain software modules to execute processes consistent with thepresent disclosure. In particular, memory device 411 may include aninput determination module 412, an output determination module 413, asensors communication module 414, a virtual content determination module415, a virtual content communication module 416, and a database accessmodule 417. Modules 412-417 may contain software instructions forexecution by at least one processor (e.g., processing device 460)associated with XR unit 204. Input determination module 412, outputdetermination module 413, sensors communication module 414, virtualcontent determination module 415, virtual content communication module416, and database access module 417 may cooperate to perform variousoperations. For example, input determination module 412 may determineUser Interface (UI) input received from input unit 202. At the sametime, sensors communication module 414 may receive data from differentsensors to determine a status of user 100. Virtual content determinationmodule 415 may determine the virtual content to display based onreceived input and the determined status of user 100. Virtual contentcommunication module 416 may retrieve virtual content not determined byvirtual content determination module 415. The retrieval of the virtualcontent may be from database 380, database 480, mobile communicationsdevice 206, or from remote processing unit 208. Based on the output ofvirtual content determination module 415, output determination module413 may cause a change in a virtual content displayed to user 100 byprojector 454.

In some embodiments, input determination module 412 may regulate theoperation of input interface 430 in order to receive gesture input 431,virtual input 432, audio input 433, and UI input 434. Consistent withthe present disclosure, input determination module 412 may concurrentlyreceive different types of input data. In one embodiment, inputdetermination module 412 may apply different rules based on the detectedtype of input. For example, gesture input may have precedence overvirtual input. In some embodiments, output determination module 413 mayregulate the operation of output interface 450 in order to generateoutput using light indicators 451, display 452, speakers 453, andprojector 454. In one embodiment, light indicators 451 may include alight indicator that shows the status of the wearable extended realityappliance. For example, the light indicator may display green light whenthe wearable extended reality appliance 110 are connected to input unit202, and blinks when wearable extended reality appliance 110 has lowbattery. In another embodiment, display 452 may be used to displayoperational information. In another embodiment, speakers 453 may includea bone conduction headphone used to output audio to user 100. In anotherembodiment, projector 454 may present virtual content to user 100.

The operations of a sensors communication module, a virtual contentdetermination module, a virtual content communication module, and adatabase access module are described above with reference to FIG. 3,details of which are not repeated herein. Modules 412-417 may beimplemented in software, hardware, firmware, a mix of any of those, orthe like.

Network interface 420, shown in FIG. 4, is assumed to have similarfunctionality as the functionality of network interface 320, describedabove in detail. The specific design and implementation of networkinterface 420 may depend on the communications network(s) over which XRunit 204 is intended to operate. For example, in some embodiments, XRunit 204 is configured to be selectively connectable by wire to inputunit 202. When connected by wire, network interface 420 may enablecommunications with input unit 202; and when not connected by wire,network interface 420 may enable communications with mobilecommunications device 206.

Input interface 430, shown in FIG. 4, is assumed to have similarfunctionality as the functionality of input interface 330 describedabove in detail. In this case, input interface 430 may communicate withan image sensor to obtain gesture input 431 (e.g., a finger of user 100pointing to a virtual object), communicate with other XR units 204 toobtain virtual input 432 (e.g., a virtual object shared with XR unit 204or a gesture of avatar detected in the virtual environment), communicatewith a microphone to obtain audio input 433 (e.g., voice commands), andcommunicate with input unit 202 to obtain UI input 434 (e.g., virtualcontent determined by virtual content determination module 315).

Power source 440, shown in FIG. 4, is assumed to have similarfunctionality as the functionality of power source 340 described above,only it provides electrical energy to power XR unit 204. In someembodiments, power source 440 may be charged by power source 340. Forexample, power source 440 may be wirelessly changed when XR unit 204 isplaced on or in proximity to input unit 202.

Output interface 450, shown in FIG. 4, is assumed to have similarfunctionality as the functionality of output interface 350 describedabove in detail. In this case, output interface 450 may cause outputfrom light indicators 451, display 452, speakers 453, and projector 454.Projector 454 may be any device, apparatus, instrument, or the likecapable of projecting (or directing) light in order to display virtualcontent onto a surface. The surface may be part of XR unit 204, part ofan eye of user 100, or part of an object in proximity to user 100. Inone embodiment, projector 454 may include a lighting unit thatconcentrates light within a limited solid angle by means of one or moremirrors and lenses, and provides a high value of luminous intensity in adefined direction.

Processing device 460, shown in FIG. 4, is assumed to have similarfunctionality as the functionality of processing device 360 describedabove in detail. When XR unit 204 is connected to input unit 202,processing device 460 may work together with processing device 360.Specifically, processing device 460 may implement virtual machinetechnologies or other technologies to provide the ability to execute,control, run, manipulate, store, etc., multiple software processes,applications, programs, etc. It is appreciated that other types ofprocessor arrangements could be implemented to provide the capabilitiesdisclosed herein.

Sensors interface 470, shown in FIG. 4, is assumed to have similarfunctionality as the functionality of sensors interface 370 describedabove in detail. Specifically sensors interface 470 may communicate withaudio sensor 471, image sensor 472, motion sensor 473, environmentalsensor 474, and other sensors 475. The operations of an audio sensor, animage sensor, a motion sensor, an environmental sensor, and othersensors are described above with reference to FIG. 3, details of whichare not repeated herein. It is appreciated that other types andcombination of sensors may be used to provide the capabilities disclosedherein.

The components and arrangements shown in FIG. 4 are not intended tolimit the disclosed embodiments. As will be appreciated by a personskilled in the art having the benefit of this disclosure, numerousvariations and/or modifications may be made to the depictedconfiguration of XR unit 204. For example, not all components may beessential for the operation of XR unit 204 in all cases. Any componentmay be located in any appropriate part of system 200, and the componentsmay be rearranged into a variety of configurations while providing thefunctionality of the disclosed embodiments. For example, some XR unitsmay not include all of the elements in XR unit 204 (e.g., wearableextended reality appliance 110 may not have light indicators 451).

FIG. 5 is a block diagram of an examplary configuration of remoteprocessing unit 208. FIG. 5 is an exemplary representation of just oneembodiment, and it is to be understood that some illustrated elementsmight be omitted and others added within the scope of this disclosure.In the embodiment of FIG. 5, remote processing unit 208 may include aserver 210 that directly or indirectly accesses a bus 500 (or othercommunication mechanism) interconnecting subsystems and components fortransferring information within server 210. For example, bus 500 mayinterconnect a memory interface 510, a network interface 520, a powersource 540, a processing device 560, and a database 580. Remoteprocessing unit 208 may also include a one or more data structures. Forexample, data structures 212A, 212B, and 212C.

Memory interface 510, shown in FIG. 5, is assumed to have similarfunctionality as the functionality of memory interface 310 describedabove in detail. Memory interface 510 may be used to access a softwareproduct and/or data stored on a non-transitory computer-readable mediumor on other memory devices, such as memory devices 311, 411, 511, ordata structures 212A, 212B, and 212C. Memory device 511 may containsoftware modules to execute processes consistent with the presentdisclosure. In particular, memory device 511 may include a shared memorymodule 512, a node registration module 513, a load balancing module 514,one or more computational nodes 515, an internal communication module516, an external communication module 517, and a database access module(not shown). Modules 512-517 may contain software instructions forexecution by at least one processor (e.g., processing device 560)associated with remote processing unit 208. Shared memory module 512,node registration module 513, load balancing module 514, computationalmodule 515, and external communication module 517 may cooperate toperform various operations.

Shared memory module 512 may allow information sharing between remoteprocessing unit 208 and other components of system 200. In someembodiments, shared memory module 512 may be configured to enableprocessing device 560 (and other processing devices in system 200) toaccess, retrieve, and store data. For example, using shared memorymodule 512, processing device 560 may perform at least one of: executingsoftware programs stored on memory device 511, database 580, or datastructures 212A-C; storing information in memory device 511, database580, or data structures 212A-C; or retrieving information from memorydevice 511, database 580, or data structures 212A-C.

Node registration module 513 may be configured to track the availabilityof one or more computational nodes 515. In some examples, noderegistration module 513 may be implemented as: a software program, suchas a software program executed by one or more computational nodes 515, ahardware solution, or a combined software and hardware solution. In someimplementations, node registration module 513 may communicate with oneor more computational nodes 515, for example, using internalcommunication module 516. In some examples, one or more computationalnodes 515 may notify node registration module 513 of their status, forexample, by sending messages: at startup, at shutdown, at constantintervals, at selected times, in response to queries received from noderegistration module 513, or at any other determined times. In someexamples, node registration module 513 may query about the status of oneor more computational nodes 515, for example, by sending messages: atstartup, at constant intervals, at selected times, or at any otherdetermined times.

Load balancing module 514 may be configured to divide the workload amongone or more computational nodes 515. In some examples, load balancingmodule 514 may be implemented as: a software program, such as a softwareprogram executed by one or more of the computational nodes 515, ahardware solution, or a combined software and hardware solution. In someimplementations, load balancing module 514 may interact with noderegistration module 513 in order to obtain information regarding theavailability of one or more computational nodes 515. In someimplementations, load balancing module 514 may communicate with one ormore computational nodes 515, for example, using internal communicationmodule 516. In some examples, one or more computational nodes 515 maynotify load balancing module 514 of their status, for example, bysending messages: at startup, at shutdown, at constant intervals, atselected times, in response to queries received from load balancingmodule 514, or at any other determined times. In some examples, loadbalancing module 514 may query about the status of one or morecomputational nodes 515, for example, by sending messages: at startup,at constant intervals, at pre-selected times, or at any other determinedtimes.

Internal communication module 516 may be configured to receive and/or totransmit information from one or more components of remote processingunit 208. For example, control signals and/or synchronization signalsmay be sent and/or received through internal communication module 516.In one embodiment, input information for computer programs, outputinformation of computer programs, and/or intermediate information ofcomputer programs may be sent and/or received through internalcommunication module 516. In another embodiment, information receivedthough internal communication module 516 may be stored in memory device511, in database 580, in data structures 212A-C, or other memory devicein system 200. For example, information retrieved from data structure212A may be transmitted using internal communication module 516. Inanother example, input data may be received using internal communicationmodule 516 and stored in data structure 212B.

External communication module 517 may be configured to receive and/or totransmit information from one or more components of system 200. Forexample, control signals may be sent and/or received through externalcommunication module 517. In one embodiment, information received thoughexternal communication module 517 may be stored in memory device 511, indatabase 580, in data structures 212A-C, and or any memory device in thesystem 200. In another embodiment, information retrieved from any ofdata structures 212A-C may be transmitted using external communicationmodule 517 to XR unit 204. In another embodiment, input data may betransmitted and/or received using external communication module 517.Examples of such input data may include data received from input unit202, information captured from the environment of user 100 using one ormore sensors (e.g., audio sensor 471, image sensor 472, motion sensor473, environmental sensor 474, other sensors 475), and more.

In some embodiments, aspects of modules 512-517 may be implemented inhardware, in software (including in one or more signal processing and/orapplication specific integrated circuits), in firmware, or in anycombination thereof, executable by one or more processors, alone, or invarious combinations with each other. Specifically, modules 512-517 maybe configured to interact with each other and/or other modules of system200 to perform functions consistent with disclosed embodiments. Memorydevice 511 may include additional modules and instructions or fewermodules and instructions.

Network interface 520, power source 540, processing device 560, anddatabase 580, shown in FIG. 5, are assumed to have similar functionalityas the functionality of similar elements described above with referenceto FIGS. 4 and 5. The specific design and implementation of theabove-mentioned components may vary based on the implementation ofsystem 200. In addition, remote processing unit 208 may include more orfewer components. For example, remote processing unit 208 may include aninput interface configured to receive direct input from one or moreinput devices.

Consistent with the present disclosure, a processing device of system200 (e.g., processor within mobile communications device 206, aprocessor within a server 210, a processor within a wearable extendedreality appliance, such as, wearable extended reality appliance 110,and/or a processor within an input device associated with wearableextended reality appliance 110, such as keyboard 104) may use machinelearning algorithms in order to implement any of the methods disclosedherein. In some embodiments, machine learning algorithms (also referredto as machine learning models in the present disclosure) may be trainedusing training examples, for example in the cases described below. Somenon-limiting examples of such machine learning algorithms may includeclassification algorithms, data regressions algorithms, imagesegmentation algorithms, visual detection algorithms (such as objectdetectors, face detectors, person detectors, motion detectors, edgedetectors, etc.), visual recognition algorithms (such as facerecognition, person recognition, object recognition, etc.), speechrecognition algorithms, mathematical embedding algorithms, naturallanguage processing algorithms, support vector machines, random forests,nearest neighbors algorithms, deep learning algorithms, artificialneural network algorithms, convolutional neural network algorithms,recurrent neural network algorithms, linear machine learning models,non-linear machine learning models, ensemble algorithms, and more. Forexample, a trained machine learning algorithm may comprise an inferencemodel, such as a predictive model, a classification model, a dataregression model, a clustering model, a segmentation model, anartificial neural network (such as a deep neural network, aconvolutional neural network, a recurrent neural network, etc.), arandom forest, a support vector machine, and so forth. In some examples,the training examples may include example inputs together with thedesired outputs corresponding to the example inputs. Further, in someexamples, training machine learning algorithms using the trainingexamples may generate a trained machine learning algorithm, and thetrained machine learning algorithm may be used to estimate outputs forinputs not included in the training examples. In some examples,engineers, scientists, processes and machines that train machinelearning algorithms may further use validation examples and/or testexamples. For example, validation examples and/or test examples mayinclude example inputs together with the desired outputs correspondingto the example inputs, a trained machine learning algorithm and/or anintermediately trained machine learning algorithm may be used toestimate outputs for the example inputs of the validation examplesand/or test examples, the estimated outputs may be compared to thecorresponding desired outputs, and the trained machine learningalgorithm and/or the intermediately trained machine learning algorithmmay be evaluated based on a result of the comparison. In some examples,a machine learning algorithm may have parameters and hyper parameters,where the hyper parameters may be set manually by a person orautomatically by an process external to the machine learning algorithm(such as a hyper parameter search algorithm), and the parameters of themachine learning algorithm may be set by the machine learning algorithmbased on the training examples. In some implementations, thehyper-parameters may be set based on the training examples and thevalidation examples, and the parameters may be set based on the trainingexamples and the selected hyper-parameters. For example, given thehyper-parameters, the parameters may be conditionally independent of thevalidation examples.

In some embodiments, trained machine learning algorithms (also referredto as machine learning models and trained machine learning models in thepresent disclosure) may be used to analyze inputs and generate outputs,for example in the cases described below. In some examples, a trainedmachine learning algorithm may be used as an inference model that whenprovided with an input generates an inferred output. For example, atrained machine learning algorithm may include a classificationalgorithm, the input may include a sample, and the inferred output mayinclude a classification of the sample (such as an inferred label, aninferred tag, and so forth). In another example, a trained machinelearning algorithm may include a regression model, the input may includea sample, and the inferred output may include an inferred valuecorresponding to the sample. In yet another example, a trained machinelearning algorithm may include a clustering model, the input may includea sample, and the inferred output may include an assignment of thesample to at least one cluster. In an additional example, a trainedmachine learning algorithm may include a classification algorithm, theinput may include an image, and the inferred output may include aclassification of an item depicted in the image. In yet another example,a trained machine learning algorithm may include a regression model, theinput may include an image, and the inferred output may include aninferred value corresponding to an item depicted in the image (such asan estimated property of the item, such as size, volume, age of a persondepicted in the image, distance from an item depicted in the image, andso forth). In an additional example, a trained machine learningalgorithm may include an image segmentation model, the input may includean image, and the inferred output may include a segmentation of theimage. In yet another example, a trained machine learning algorithm mayinclude an object detector, the input may include an image, and theinferred output may include one or more detected objects in the imageand/or one or more locations of objects within the image. In someexamples, the trained machine learning algorithm may include one or moreformulas and/or one or more functions and/or one or more rules and/orone or more procedures, the input may be used as input to the formulasand/or functions and/or rules and/or procedures, and the inferred outputmay be based on the outputs of the formulas and/or functions and/orrules and/or procedures (for example, selecting one of the outputs ofthe formulas and/or functions and/or rules and/or procedures, using astatistical measure of the outputs of the formulas and/or functionsand/or rules and/or procedures, and so forth).

Consistent with the present disclosure, a processing device of system200 may analyze image data captured by an image sensor (e.g., imagesensor 372, image sensor 472, or any other image sensor) in order toimplement any of the methods disclosed herein. In some embodiments,analyzing the image data may comprise analyzing the image data to obtaina preprocessed image data, and subsequently analyzing the image dataand/or the preprocessed image data to obtain the desired outcome. One ofordinary skill in the art will recognize that the followings areexamples, and that the image data may be preprocessed using other kindsof preprocessing methods. In some examples, the image data may bepreprocessed by transforming the image data using a transformationfunction to obtain a transformed image data, and the preprocessed imagedata may comprise the transformed image data. For example, thetransformed image data may comprise one or more convolutions of theimage data. For example, the transformation function may comprise one ormore image filters, such as low-pass filters, high-pass filters,band-pass filters, all-pass filters, and so forth. In some examples, thetransformation function may comprise a nonlinear function. In someexamples, the image data may be preprocessed by smoothing at least partsof the image data, for example using Gaussian convolution, using amedian filter, and so forth. In some examples, the image data may bepreprocessed to obtain a different representation of the image data. Forexample, the preprocessed image data may comprise: a representation ofat least part of the image data in a frequency domain; a DiscreteFourier Transform of at least part of the image data; a Discrete WaveletTransform of at least part of the image data; a time/frequencyrepresentation of at least part of the image data; a representation ofat least part of the image data in a lower dimension; a lossyrepresentation of at least part of the image data; a losslessrepresentation of at least part of the image data; a time ordered seriesof any of the above; any combination of the above; and so forth. In someexamples, the image data may be preprocessed to extract edges, and thepreprocessed image data may comprise information based on and/or relatedto the extracted edges. In some examples, the image data may bepreprocessed to extract image features from the image data. Somenon-limiting examples of such image features may comprise informationbased on and/or related to: edges; corners; blobs; ridges; ScaleInvariant Feature Transform (SIFT) features; temporal features; and soforth. In some examples, analyzing the image data may includecalculating at least one convolution of at least a portion of the imagedata, and using the calculated at least one convolution to calculate atleast one resulting value and/or to make determinations,identifications, recognitions, classifications, and so forth.

Consistent with other aspects of the disclosure, a processing device ofsystem 200 may analyze image data in order to implement any of themethods disclosed herein. In some embodiments, analyzing the image maycomprise analyzing the image data and/or the preprocessed image datausing one or more rules, functions, procedures, artificial neuralnetworks, object detection algorithms, face detection algorithms, visualevent detection algorithms, action detection algorithms, motiondetection algorithms, background subtraction algorithms, inferencemodels, and so forth. Some non-limiting examples of such inferencemodels may include: an inference model preprogrammed manually; aclassification model; a regression model; a result of trainingalgorithms, such as machine learning algorithms and/or deep learningalgorithms, on training examples, where the training examples mayinclude examples of data instances, and in some cases, a data instancemay be labeled with a corresponding desired label and/or result, andmore. In some embodiments, analyzing image data (for example by themethods, steps and modules described herein) may comprise analyzingpixels, voxels, point cloud, range data, etc. included in the imagedata.

A convolution may include a convolution of any dimension. Aone-dimensional convolution is a function that transforms an originalsequence of numbers to a transformed sequence of numbers. Theone-dimensional convolution may be defined by a sequence of scalars.Each particular value in the transformed sequence of numbers may bedetermined by calculating a linear combination of values in asubsequence of the original sequence of numbers corresponding to theparticular value. A result value of a calculated convolution may includeany value in the transformed sequence of numbers. Likewise, ann-dimensional convolution is a function that transforms an originaln-dimensional array to a transformed array. The n-dimensionalconvolution may be defined by an n-dimensional array of scalars (knownas the kernel of the n-dimensional convolution). Each particular valuein the transformed array may be determined by calculating a linearcombination of values in an n-dimensional region of the original arraycorresponding to the particular value. A result value of a calculatedconvolution may include any value in the transformed array. In someexamples, an image may comprise one or more components (such as colorcomponents, depth component, etc.), and each component may include atwo-dimensional array of pixel values. In one example, calculating aconvolution of an image may include calculating a two-dimensionalconvolution on one or more components of the image. In another example,calculating a convolution of an image may include stacking arrays fromdifferent components to create a three-dimensional array, andcalculating a three dimensional convolution on the resulting threedimensional array. In some examples, a video may comprise one or morecomponents (such as color components, depth component, etc.), and eachcomponent may include a three-dimensional array of pixel values (withtwo spatial axes and one temporal axis). In one example, calculating aconvolution of a video may include calculating a three-dimensionalconvolution on one or more components of the video. In another example,calculating a convolution of a video may include stacking arrays fromdifferent components to create a four-dimensional array, and calculatinga four dimensional convolution on the resulting four dimensional array.

In some embodiments, an integrated computational interface device mayinclude a portable housing having a key region and a non-key region. Ahousing of the integrated computational interface device may include anouter covering or shell that may include one or more componentsassociated with the integrated computational interface device. Thedisclosed exemplary housing may surround the components of theintegrated computational interface device and may cover some or allcomponents of the integrated computational interface device. It iscontemplated that the disclosed exemplary housing may have one or moreopenings that may expose certain components (e.g., USB or other ports)of the integrated computational interface device or may allow certaincomponents (e.g., keys of a keyboard) to protrude from the housing. Thehousing may support certain components of the integrated computationalinterface device (e.g., a circuit board) in an interior portion of thehousing. The housing may include additional structural features that maypermit one or more components of the integrated computational interfacedevice to be attached to the housing. The housing may be square,rectangular, or other shape sized to fit on a user's desk, lap, or othersuitable work surface. The housing may be made of plastic, metal, acombination of plastic and metal, or other suitable material.

The housing of the integrated computational device may include a keyregion and a non-key region distinct from the key region. The key regionof the integrated computational interface device may include one or morekeys that may allow a user to enter alphanumeric or other characters asinputs. For example, in some embodiments, a keyboard may be associatedwith the key region of the housing. The keyboard may be a standardtypewriter-style keyboard (e.g., a QWERTY-style keyboard) or othersuitable keyboard layout, such as a Dvorak layout or a chorded layout.The keyboard may include any suitable number of keys; for example, a“full size” keyboard may include up to 104 or 105 keys. In someembodiments, the keyboard may include at least 30 keys. In otherembodiments, the keyboard may include less than ten keys, at least tenkeys, at least 20 keys, at least 50 keys, at least 80 keys, at least 100keys, and so forth.

The key region may include alphanumeric keys, function keys, modifierkeys, cursor keys, system keys, multimedia control keys, or otherphysical keys that perform a computer-specific function when pressed.The key region may also include virtual or programmable keys, such thatthe function of the key changes depending on a function or applicationto be performed on the integrated computational interface device.

In some embodiments, the keyboard may include a dedicated input key forexecuting actions by the wearable extended reality appliance. Thededicated input key may permit a user to interact with a virtual widgetviewed through the wearable extended reality appliance. The dedicatedinput key may take a picture of one or more virtual widgets viewedthrough the wearable extended reality appliance (i.e., a “screenshot”).If the wearable extended reality appliance includes a camera, thededicated input key may take a picture by the camera. In an embodiment,the picture may include what the user sees through the wearable extendedreality device with any virtual widgets included in the picture (i.e., apicture with virtual overlay). The keyboard may include multiplededicated input keys for executing actions, with each key configured toperform a different action. In an embodiment, the dedicated input keysmay be programmable by the user to perform one or more actions (e.g., a“macro”). It is noted that the above examples of actions executed by thededicated input key are not limiting and other actions may be performedby the dedicated input key.

In some embodiments, the keyboard may include a dedicated input key forchanging illumination of a virtual display projected by the wearableextended reality appliance. The illumination of the virtual display maybe adjusted by turning the virtual display on or off or by increasing ordecreasing the brightness, contrast, or other color settings of thevirtual display. In an embodiment, the keyboard may include multiplededicated input keys for adjusting different settings of the virtualdisplay. The settings of the virtual display that may be adjusted mayinclude, for example: picture settings, such as brightness, contrast,sharpness, or display mode (e.g., a game mode with predefined settings);color settings, such as color component levels or other color adjustmentsettings; a position of the virtual display relative to a location ofthe user's head; or other settings that may enhance the user's view ofthe virtual display.

The non-key region of the portable housing may be a region of thehousing that does not include any keys. The non-key region may be aregion of the housing that does not include any keys and may exist tocomplete a desired shape of the housing that extends beyond the keyregion of the housing in any direction. The non-key region may be aregion that may include an input device, such as a track pad, atouchscreen, a touch bar, or other form of cursor control for theintegrated computational interface device. The non-key region may besubdivided into multiple different non-key regions, such as the trackpad or other cursor control, an extension of the housing, or a cover orgrille for one or more speakers or other audio output devices includedwithin the housing. The non-key region may include a display fordisplaying information to the user, may include one or more openings topermit air to circulate through the housing (e.g., for coolingcomponents contained within the housing), or may include one or moredoors or access ports to permit access to an interior portion of thehousing (e.g., a battery compartment configured to retain a removablebattery therein or to permit certain components to be installed intoand/or removed from the interior portion of the housing).

In some embodiments, a holder may be associated with the non-key regionof the housing. The holder may be a depression that extends below asurface of the housing or may rest entirely above the surface of thehousing. The holder may be located in any portion of the non-key regionof the housing. In some embodiments, the holder may be located in anon-key region of the housing adjacent to the key region of the housing.The holder may include one or more structural features to selectivelyengage with one or more items such as writing instruments, wires orcables, dongles, papers (i.e., to permit the holder to function like acopy stand), or other items that the user may wish to easily access orstore.

In some embodiments, the holder may be configured for selectiveengagement with and disengagement from a wearable extended realityappliance, such that when the wearable extended reality appliance isselectively engaged with the housing via the holder, the wearableextended reality appliance is transportable with the housing. In someembodiments, the structural features of the holder may be configured toselectively engage with a wearable extended reality appliance such thatthe wearable extended reality appliance may be snap-fit, force-fit, orpressure-fit into at least a portion of the holder. When the wearableextended reality appliance is selectively engaged with the housing viathe holder, the wearable extended reality appliance may be transportablewith the housing by being securely connected to the housing via theholder.

FIG. 6 is a top view of an exemplary embodiment of an integratedcomputational interface device 610 with a wearable extended realityappliance 612 selectively engaged with integrated computationalinterface device 610. FIG. 6 is an exemplary representation of just oneembodiment, and it is to be understood that some illustrated elementsmight be omitted and others added within the scope of this disclosure.

Integrated computational interface device 610 may include housing 614.Housing 614 may include key region 616 and non-key regions 618 a and 618b. As shown in FIG. 6, non-key region 618 a may include a regionpositioned above key region 616 and non-key region 618 b may include aregion positioned below key region 616. Housing 614 may include only onenon-key region (e.g., only either non-key region 618 a or non-key region618 b), may have non-key region 618 a and non-key region 618 b locatednext to each other, or may have more than two non-key regions.

Key region 616 may include a keyboard 620. Key region 616 may besubdivided into multiple key regions, and the key regions may becontiguous or may be separated from each other.

Integrated computational interface device 610 may include holder 622that may be configured for selective engagement with wearable extendedreality appliance 612. In some embodiments, wearable extended realityappliance may include a pair of smart glasses. As shown in FIG. 6,wearable extended reality appliance 612 may be a pair of smart glasses.Smart glasses may appear similar to traditional eyeglasses and mayinclude “smart” functionality such as a camera positioned to takepictures of what the user is presently viewing or one or more displaysconfigured to project images onto lenses of the smart glasses. In someembodiments, wearable extended reality appliance 612 may take otherforms that may include one or more lenses, such as goggles or other formof wearable device.

In some embodiments, integrated computational interface device mayinclude a track pad associated with the housing. The track pad mayenable the user of the integrated computational interface device tocontrol a cursor, select items, or activate items on the integratedcomputational interface device. The track pad may include a singlesurface or a segmented surface, such as a cursor control portion and oneor more button portions.

In an embodiment where the wearable extended reality appliance is a pairof smart glasses, the integrated computational interface device isconfigured such that when the pair of smart glasses is selectivelyengaged with the housing via the holder, temples of the smart glassescontact the track pad. Thus, for example, the holder may be located at afirst portion of the housing and the track pad may be located at asecond portion of the housing, the first portion of the housing beingspaced apart from the second portion of the housing. The holder may bespaced apart from the track pad by a distance approximately equal to thelength of the temple portion of the pair of smart glasses.

The temples of the pair of smart glasses may each include an elastictrack pad protector on a distal end thereof. The temples may extend awayfrom the lenses of the smart glasses in one direction and in parallel toeach other to enable the smart glasses to be worn by the user. In someexamples, the pair of smart glasses may include at least two temples andat least one lens. Each temple may include a temple tip, and each templetip may include an elastic track pad protector outer part. In someexamples, the pair of smart glasses may include at least two temples andat least one lens, each temple may include an elastic track padprotector. The track pad protector may protect the distal ends thetemples of the smart glasses from being damaged when the smart glassesare selectively engaged with the holder and the distal ends of thetemples are proximate to the surface of the housing (e.g., the trackpad). The track pad protector may include a sleeve that slides over thedistal end of the temple or may be integrally formed with the distal endof the temple. The track pad protector may be made of a soft or flexiblematerial such that the distal end of the temples of the pair of smartglasses does not scratch or damage the track pad when the pair of smartglasses is selectively engaged with the housing via the holder and thedistal end of the temples of the pair of smart glasses contacts thetrack pad.

FIG. 7A is a top view and FIG. 7B is a left side view of a secondexemplary embodiment of integrated computational interface device 710with a wearable extended reality appliance in the form of a pair ofsmart glasses 712 selectively engaged with integrated computationalinterface device 710. FIGS. 7A and 7B are exemplary representations ofjust one embodiment, and it is to be understood that some illustratedelements might be omitted and others added within the scope of thisdisclosure.

Integrated computational interface device 710 may include housing 714.Housing 714 may include key region 716 and non-key regions 718 a and 718b. Key region 716 may include a keyboard 720. Integrated computationalinterface device 710 may include holder 722 that is configured forselective engagement with the pair of smart glasses 712. Non-key region718 b may be positioned below keyboard 720 and may include track pad724. Integrated computational interface device 710 shown in FIGS. 7A and7B may have similar structural and functional characteristics asintegrated computational interface device 610 shown in FIG. 6. Forexample, integrated computational interface device 710 may include allor some of the elements of integrated computational interface device610.

The pair of smart glasses 712 may include lens portion 726 and twotemples 728, one temple on each end of lens portion 726. Each temple 728may include proximal end 730 that connects temple 728 to lens portion726 and distal end 732 located at an opposite end of temple 728. Distalend 732 may include elastic track pad protector 734. As noted above,elastic track pad protector 734 may be positioned over distal end 732 ormay be integrally formed with distal end 732.

In some embodiments, the holder of the integrated computationalinterface device may include at least two grasping elements configuredfor selective engagement with the temples of the pair of smart glasses.In some embodiments, the grasping elements may be configured forselective engagement with different parts of the smart glasses, such asone or both lenses, a bridge between the lenses, or other portions ofthe smart glasses. The grasping elements may be integrally formed withthe holder, may be individually removable from the holder, or may bejointly removable from the holder. For example, the grasping elementsmay be integrally formed with each other and may be removable as a unitfrom the holder. The grasping elements may be spring-biased towards eachother, either by virtue of their shape or by using a spring orspring-like member.

Each grasping element may include a protuberance from a surface of theintegrated computational interface device. The protuberance may extendperpendicularly away from the surface of the integrated computationalinterface device. A depression may be formed within the protuberance andmay be configured to hold the temple. The depression may be in a side ofthe protuberance opposite to the surface of the integrated computationalinterface device, such that the temple is positioned on a top surface ofthe protuberance. The depression may be in a side of the protuberanceparallel to the surface of the integrated computational interfacedevice, such that the temple is positioned on a side surface of theprotuberance. The holder may be made of a flexible material, a rigid orsemi-rigid material, or a rigid or semi-rigid material with a flexibledepression. The flexible depression may be integrally formed with theprotuberance or may be a flexible material covering the depression.

FIG. 8A is a front perspective view of a wearable extended realityappliance selectively engaged with a first embodiment of a holder. FIG.8B is a rear perspective view of the wearable extended reality applianceselectively disengaged from the first embodiment of the holder shown inFIG. 8A. FIGS. 8A and 8B are an exemplary representation of just oneembodiment, and it is to be understood that some illustrated elementsmight be omitted and others added within the scope of this disclosure.

Holder 810 may include two grasping elements 812 spaced apart from eachother and configured to selectively engage temple 814 of wearableextended reality appliance 816, which is shown in FIGS. 8A and 8B as apair of smart glasses. In one example, holder 810 may be part of holder622 and/or of holder 722 and/or of holder 1218 a. In some embodiments,there may be other configurations of grasping elements 812, such as onlyone grasping element 812 or the two grasping elements 812 may be locatedon a same side of holder 810.

In some embodiments, each grasping element 812 may include depression818 on a top surface to engage temple 814. In some embodiments, eachgrasping element 812 may have a flat top surface to engage temple 814.As shown in FIG. 8B, depression 818 may be U-shaped to partiallysurround temple 814. In some embodiments, depression 818 may be shapeddifferently to engage temple 814.

In some embodiments, the holder of the integrated computationalinterface device may include a clip for selectively connecting thewearable extended reality appliance with the housing. Selectivelyconnecting the wearable extended reality appliance with the housing isone example of selectively engaging the wearable extended realityappliance with the housing. The clip may be positioned in any portion ofthe holder and may include a protuberance from a surface of theintegrated computational interface device. The clip may selectivelyengage with any portion of the wearable extended reality appliance toconnect the wearable extended reality appliance to the holder. In anembodiment where the wearable extended reality appliance is a pair ofsmart glasses, the clip may selectively engage with a temple, a portionof a lens, a portion of a rim surrounding a lens, a bridge, or a nosepad. The holder may include additional features to selectively engagewith other portions of the wearable extended reality appliance that arenot selectively engaged by the clip. The additional features may includeone or more additional protuberances extending from a surface of theholder. The holder may be made of a flexible material, a rigid orsemi-rigid material, or a rigid or semi-rigid material with a flexibleclip or a flexible protuberance. The flexible clip or the flexibleprotuberance may be integrally formed with the holder, may be detachablefrom the holder, or may be a flexible material covering the clip or theprotuberance.

FIG. 9A is a front perspective view of a wearable extended realityappliance selectively engaged with a second exemplary embodiment of aholder. FIG. 9B is a rear perspective view of the wearable extendedreality appliance selectively disengaged from the second embodiment ofthe holder shown in FIG. 9A. FIGS. 9A and 9B are an exemplaryrepresentation of just one embodiment, and it is to be understood thatsome illustrated elements might be omitted and others added within thescope of this disclosure.

Holder 910 may include clip 912 configured to selectively engage bridge914 of wearable extended reality appliance 916. As shown in FIGS. 9A and9B, wearable extended reality appliance 916 may include a pair of smartglasses. In one example, holder 910 may be part of holder 622 and/or ofholder 722 and/or of holder 1218 a. In one exemplary embodiment asillustrated in FIG. 9B, clip 912 may include post 918 configured to fitbetween nose pads 920 of wearable extended reality appliance 916. Asshown in FIG. 9B, post 918 may have a round or cylindrical shape. Post918 may be in other shapes (e.g., having a square, rectangular,elliptical, or polygonal cross-section) such that nose pads 920 ofwearable extended reality appliance 916 fit around post 918 toselectively engage post 918.

Clip 912 may include bridge protuberance 922 configured to contact afront portion of bridge 914. Bridge protuberance 922 may be spaced apartfrom post 918 such that a portion of bridge 914 is located betweenbridge protuberance 922 and post 918 when wearable extended realityappliance 916 is selectively engaged with holder 910.

Holder 910 may include at least two lens protuberances 924. Lensprotuberances 924 are spaced apart from clip 912 such that each lensprotuberance 924 selectively engages an outer portion of a lens 926 ofwearable extended reality appliance 916. As shown in FIG. 9A, bridgeprotuberance 922 may be shaped such that an inner portion of lenses 926of wearable extended reality appliance 916 selectively engage bridgeprotuberance 922.

In some embodiments, clip 912 may only include post 918. In someembodiments, holder 910 may include only clip 912 and not lensprotuberances 924. In some embodiments, holder 910 may include only lensprotuberances 924 and not clip 912.

In some embodiments, the holder of the integrated computationalinterface device may include a compartment for selectively enclosing atleast a portion of the wearable extended reality appliance when thewearable extended reality appliance is selectively engaged with theholder. The compartment may include a depression or sleeve in thehousing to receive one or more portions of the wearable extended realityappliance. The compartment may include a protuberance from a surface ofthe holder and may be shaped such that the wearable extended realityappliance does not slide front-to-back or side-to-side within thecompartment when the wearable extended reality appliance is selectivelyengaged with the holder. The protuberances may include one or more wallsextending above the surface of the holder such that the walls enclose aportion of the wearable extended reality appliance. The walls may beconfigured to accommodate different shapes of the wearable extendedreality appliance. For example, if the wearable extended realityappliance is a pair of smart glasses, the walls may include a cut-outportion such that a nose pad of the smart glasses does not contact thewalls. The walls may also taper towards the surface of the holder toaccommodate the lenses of the smart glasses such that a bottom portionof the lenses contacts the surface of the holder. The holder may be madeof a flexible material, a rigid or semi-rigid material, or a rigid orsemi-rigid material with a flexible compartment. The flexiblecompartment may be integrally formed with the holder or may be aflexible material covering the compartment.

FIG. 10A is a front perspective view of a wearable extended realityappliance selectively engaged with a third exemplary embodiment of aholder. FIG. 10B is a rear perspective view of the wearable extendedreality appliance selectively disengaged from the third exemplaryembodiment of the holder shown in FIG. 10A. FIGS. 10A and 10B are anexemplary representation of just one embodiment, and it is to beunderstood that some illustrated elements might be omitted and othersadded within the scope of this disclosure.

Holder 1010 may include compartment 1012 configured to selectivelyenclose at least a portion of wearable extended reality appliance 1014.In one example, holder 1010 may be part of holder 622 and/or of holder722 and/or of holder 1218 a. Compartment 1012 may include wall 1016 thatextends along a portion a surface of holder 1010 such that wall 1016defines an outer periphery of compartment 1012. As shown in FIG. 10A,wall 1016 may taper toward a center of holder 1010 and the surface ofholder 1010 such that a lens 1018 of wearable extended reality appliance1014 contacts the surface of holder 1010 when wearable extended realityappliance 1014 is selectively engaged with holder 1010. Based on thisexemplary shape, wall 1016 may contact a user-facing side of lens 1018and not an outward-facing side of lens 1018. In some embodiments, wall1016 may not taper such that wall 1016 is of approximately even heighton all sides, to define a slot configured to receive lens 1018. As shownin FIG. 10B, wall 1016 may include cut-out portion 1020 to accommodatenose pads 1022 of wearable extended reality appliance 1014 such thatnose pads 1022 do not contact wall 1016.

In some embodiments, the holder of the integrated computationalinterface device may include at least one indentation corresponding to ashape of a portion of the wearable extended reality appliance. In anembodiment where the wearable extended reality appliance is a pair ofsmart glasses, the holder may include one or more indentations that maybe shaped to correspond to shapes of the lenses of the smart glasses. Inan embodiment where the wearable extended reality appliance is a goggle,the one or more indentations may be shaped to correspond to a shape ofthe lens of the goggle. The one or more indentations may extend below asurface of the holder such that the one or more indentations extend intoa portion of the housing. A bottom portion of the one or moreindentations may contact a surface of the housing such that at least aportion of the holder extends above the surface of the housing. Theholder may be made of a flexible material, a rigid or semi-rigidmaterial, or a rigid or semi-rigid material with a flexible indentation.The flexible indentation may be integrally formed with the holder or maybe a flexible material covering the indentation.

FIG. 11A is a front perspective view of a wearable extended realityappliance selectively engaged with a fourth exemplary embodiment of aholder. FIG. 11B is a rear perspective view of the wearable extendedreality appliance selectively disengaged from the fourth exemplaryembodiment of the holder shown in FIG. 11A. FIGS. 11A and 11B are anexemplary representation of just one embodiment, and it is to beunderstood that some illustrated elements might be omitted and othersadded within the scope of this disclosure.

Holder 1110 may include indentation 1112 corresponding to a shape of aportion of wearable extended reality appliance 1114. In one example,holder 1110 may be part of holder 622 and/or of holder 722 and/or ofholder 1218 a. As shown in FIGS. 11A and 11B, wearable extended realityappliance 1114 may include a pair of smart glasses and there may be twoindentations 1112 spaced apart from each other in holder 1110, eachindentation 1112 corresponding to a shape of lens 1116 of the smartglasses.

In some embodiments, the holder may also include a nose bridgeprojection. In an embodiment where the wearable extended realityappliance includes a pair of extended reality glasses, the at least oneindentation may include two indentations on opposite sides of the nosebridge projection, to receive lenses of the extended reality glasses. Itis noted that the terms “extended reality glasses” and “smart glasses”may be used interchangeably herein. The nose bridge projection may be aprotuberance extending away from a surface of the holder and may beconfigured to support nose pads or a bridge of the smart glasses. Thetwo indentations may each be shaped to receive one lens, the rimsurrounding the lens, or a portion of the frame surrounding the lens.The holder may be made of a flexible material, a rigid or semi-rigidmaterial, or a rigid or semi-rigid material with a flexible nose bridgeprojection. The flexible nose bridge projection may be integrally formedwith the holder or may be a flexible material covering the nose bridgeprojection.

Referring back to FIGS. 11A and 11B, wearable extended reality appliance1114 is a pair of smart glasses. Holder 1110 may include nose bridgeprojection 1118 configured to support nose pads 1120 or bridge 1122 ofthe smart glasses.

In an embodiment where the wearable extended reality appliance includesa pair of smart glasses, the holder may be configured such that whenlenses of the smart glasses are located on one side of the keyboard,temples of the smart glasses extend over the keyboard with distal endsthereof located on a side of the keyboard opposite the lenses. Theholder may include features to help position temples of the smartglasses above a surface of the housing. The features may includeprotuberances extending upward from the housing to engage the temples ofthe smart glasses. The protuberances may be located in the key region orthe non-key region of the housing.

In some embodiments, the holder may include a protuberance located nearthe keyboard (e.g., between the holder and the key region) and may beconfigured to create a gap between temples of the smart glasses and thekeyboard such that when the smart glasses are selectively engaged withthe holder, the temples of the smart glasses extend over the keyboardwith distal ends of the temples located on a side of the keyboardopposite the lenses. The distal ends of the temples may not contact thehousing because the protuberance may elevate the distal ends above asurface of the housing. The protuberance may be made of elastic or othercompressible material such that when the temples of the smart glassescontact the protuberance, the temples are not damaged or scratched.

FIG. 12A is a top view and FIG. 12B is a left side view of a thirdembodiment of an integrated computational interface device 1210 with awearable extended reality appliance in the form of a pair of smartglasses 1212 selectively engaged with the integrated computationalinterface device 1210. FIGS. 12A and 12B are exemplary representationsof just one embodiment, and it is to be understood that some illustratedelements might be omitted and others added within the scope of thisdisclosure. Integrated computational interface device 1210 shown inFIGS. 12A and 12B may have similar structural and functionalcharacteristics as integrated computational interface device 610 shownin FIG. 6 and/or as integrated computational interface device 710 shownin FIGS. 7A and 7B. For example, integrated computational interfacedevice 1210 may include all or some of the elements of integratedcomputational interface device 610. In another example, integratedcomputational interface device 1210 may include all or some of theelements of integrated computational interface device 710.

Integrated computational interface device 1210 may include housing 1214.Housing 1214 may include keyboard 1216 and holder 1218 a that may beconfigured for selective engagement with smart glasses 1212. Holder 1218a may include protuberance 1220 located near keyboard 1216. When lenses1222 of smart glasses 1212 are selectively engaged with holder 1218 a,temples 1224 may contact protuberance 1220 to thereby create gap 1226between temples 1224 and keyboard 1216. When lenses 1222 of smartglasses 1212 are selectively engaged with holder 1218 a, temples 1224may extend over keyboard 1216 such that distal ends 1228 of temples 1224are located on a side of keyboard 1216 opposite lenses 1222. Distal ends1228 may be spaced apart from housing 1214 such that distal ends 1228 donot contact housing 1214 when smart glasses 1212 are selectively engagedwith holder 1218 a.

In some embodiments, the integrated computational interface device mayfurther include a charger associated with the housing. The charger maybe configured to charge the wearable extended reality appliance when thewearable extended reality appliance is selectively engaged with theholder. In such embodiments, the wearable extended reality appliance mayinclude a battery or other power source to be charged. The charger maysupply a DC voltage or current to charge the battery of the wearableextended reality appliance. The battery may be charged by a wiredconnection or by a wireless connection. The housing and the wearableextended reality appliance may be configured for wireless charging ofthe wearable extended reality appliance by the charger. The charger maybe located in any suitable portion of the housing such that the chargercan supply power to the wearable extended reality appliance when thewearable extended reality appliance is selectively engaged with theholder.

The wearable extended reality appliance may include one or moreelectrical contacts and the housing may include one or morecorresponding electrical contacts to engage to charge the wearableextended reality appliance when engaged with the holder. In someexamples, the one or more electrical contacts included in the wearableextended reality appliance may be located in one or both of the lenses,one or both of the temples, or a portion of the frame near where thelenses and the temples connect. In some examples, the one or morecorresponding electrical contacts included in the housing may be locatedin the holder or in the housing adjacent to the holder such that whenthe wearable extended reality appliance is selectively engaged with theholder, the one or more electrical contacts of the wearable extendedreality appliance are sufficiently close that wireless charging mayoccur. The wireless charging may be performed by a wireless chargingstandard, such as Qi, AirFuel Resonant, near-field magnetic coupling(NFMC), radio frequency (RF), or other suitable wireless chargingprotocol. In some embodiments, the number of electrical contactsincluded in the housing does not have to match the number of electricalcontacts included in the wearable extended reality appliance.

In an embodiment where the wearable extended reality appliance is a pairof smart glasses, each lens may include an electrical contact and thehousing may include one or more corresponding electrical contacts. Whenthe smart glasses are selectively engaged with the housing, theelectrical contacts in the lenses may be positioned sufficiently closeto the one or more corresponding electrical contacts in the housing tocomplete a wireless charging circuit.

For example, in the embodiment shown in FIGS. 8A and 8B, the electricalcontacts may be located in temples 814 of smart glasses 816.Corresponding electrical contacts may be located in grasping elements812 of holder 810.

As another example, in the embodiment shown in FIGS. 9A and 9B, theelectrical contacts may be located in bridge 914 and/or nose pads 920 ofwearable extended reality appliance 916. Corresponding electricalcontacts may be located in clip 912 and/or post 918 of holder 910. Inanother implementation of the embodiment shown in FIGS. 9A and 9B, theelectrical contacts may be located in or around lenses 926 of wearableextended reality appliance 916. Corresponding electrical contacts may belocated in lens protuberances 924 and/or in a portion of holder 910under lenses 926.

As another example, in the embodiment shown in FIGS. 10A and 10B, theelectrical contacts may be located in or around lenses 1018 of wearableextended reality appliance 1014. Corresponding electrical contacts maybe located in wall 1016.

As another example, in the embodiment shown in FIGS. 11A and 11B, theelectrical contacts may be located in or around lenses 1116 of wearableextended reality appliance 1114. Corresponding electrical contacts maybe located in indentations 1112 of holder 1110. In anotherimplementation of the embodiment shown in FIGS. 11A and 11B, theelectrical contacts may be located in nose pads 1120 or bridge 1122 ofwearable extended reality appliance 1114. Corresponding electricalcontacts may be located in nose bridge projection 1118 of holder 1110.

In some embodiments, the housing may also include a wire port configuredto receive a wire extending from the wearable extended realityappliance. The wire may be any type of wire suitable for providing powerand/or data between the integrated computation interface device and thewearable extended reality appliance. For example, the wire may be aUniversal Serial Bus (USB) type wire with appropriate connectors for thewire port and the wearable extended reality appliance if the wire isdetachable from the wearable extended reality appliance. The wire portmay be located on any part of the housing that is readily accessible toa user of the integrated computation interface device. The wire mayextend from any part of the wearable extended reality appliance. Thewire may be fixedly attached to the wearable extended reality applianceor may be detachable from the wearable extended reality appliance. In anembodiment where the wearable extended reality appliance is a pair ofsmart glasses, the wire may extend from the temple. The wire may belocated at any point along a length of the temple such that the wire maynot interfere with a user's vision or may not affect the user's abilityto wear the smart glasses.

In some embodiments, the wire port may be located at a front side of theintegrated computational interface device configured to face the userwhile the user types on the keyboard. In one example, the wire port maybe located substantially at a center of the front side of the integratedcomputational interface device (for example, less than 1 cm from thecenter, less than 2 cm from the center, less than 4 cm from the center,less than 8 cm from the center, and so forth). In another example, thewire port may be located away from the center. In another example, thewire port may be located at a side of the front side of the integratedcomputational interface device (such as a left side, a right side,etc.), for example less than 1 cm from the side edge of the front side,less than 2 cm from the side edge, less than 4 cm from the side edge,less than 8 cm from the side edge, and so forth.

For example, in the embodiment shown in FIG. 6, wire port 624 may belocated in housing 614. Wire port 624 may be located in non-key region618 b near a user of integrated computation interface device 610 whilethe user types on keyboard 620. Wire 626 may be connected to wearableextended reality appliance 612 and may be received by wire port 624.Wire 626 may be optional and is shown in dashed outline in FIG. 6. Insome examples, wire 626 may be selectively attached and detached fromwearable extended reality appliance 612. In other examples, wire 626 maybe permanently connected to wearable extended reality appliance 612. Insome examples, wire 626 may be selectively attached and detached fromwire port 624. In other examples, wire 626 may be permanently connectedto wire port 624. In some examples, wire 626 may be completely or partlyretracted into housing 610 and/or into a compartment created by housing610. When wire 626 is in a complete or a partial retracted state, wire626 may be pulled out of housing 610 and/or out of the compartment, forexample by a user.

As another example, in the embodiment shown in FIGS. 7A and 7B, wireport 736 may be located in housing 714. Wire port 736 may be located innon-key region 718 b near a user of integrated computation interfacedevice 710 while the user types on keyboard 720. Wire 738 may beconnected to smart glasses 712 and may be received by wire port 736.Locating wire port 736 in non-key region 718 b may permit the user totype on keyboard 720 while wire 738 is connected to wire port 736. Wire738 is optional and is shown in dashed outline in FIGS. 7A and 7B. Insome examples, wire 738 may be selectively attached and detached fromwearable extended reality appliance 712. In other examples, wire 738 maybe permanently connected to wearable extended reality appliance 712. Insome examples, wire 738 may be selectively attached and detached fromwire port 736. In other examples, wire 738 may be permanently connectedto wire port 736. In some examples, wire 738 may be completely or partlyretracted into housing 710 and/or into a compartment created by housing710. When wire 738 is in a complete or a partial retracted state, wire738 may be pulled out of housing 710 and/or out of the compartment, forexample by a user.

In some embodiments, the wire may be configured to electrically chargethe wearable extended reality appliance when the wire is connected tothe wire port. The wire may be any type of wire suitable for providingpower to the wearable extended reality appliance. For example, the wiremay be a Universal Serial Bus (USB) type wire with appropriateconnectors for the wire port and the wearable extended realityappliance.

In some embodiments, the integrated computational interface device mayalso include at least one processor located in the housing. The wire maybe configured to enable digital data communication between the wearableextended reality appliance and the at least one processor. The processormay include any processing device suitable for digital datacommunication. In addition to enabling digital data communication, theprocessing device may be configured to execute computer programs on theintegrated computational interface device. The wire may be any type ofwire suitable to enable digital data communication between the wearableextended reality appliance and the at least one processor. For example,the wire may be a Universal Serial Bus (USB) type wire with appropriateconnectors for the wire port and the wearable extended realityappliance.

In some embodiments, the integrated computational interface device mayalso include a processor located in the housing. The processor may beconfigured to wirelessly pair with the wearable extended realityappliance. Wireless pairing is a process to wirelessly link theintegrated computational interface device and the wearable extendedreality appliance, to enable wireless data communication between theintegrated computational interface device and the wearable extendedreality appliance. The processor may include any processing devicesuitable for implementing a wireless pairing protocol between theintegrated computational interface device and the wearable extendedreality appliance. The wireless pairing protocol may be WiFi (IEEE802.11-based), radio frequency (RF, such as ZigBee or ZWave), radiofrequency identification (RFID, such as Active Reader Passive Tag orActive Reader Active Tag), Bluetooth, Near Field Communication (NFC), orany other wireless pairing protocol usable for short-rangecommunication. The integrated computational interface device may includea visual marker adjacent to the keyboard to facilitate wireless pairingwith the wearable extended reality appliance. The visual marker may helpensure that the wearable extended reality appliance is within wirelesscommunication range of the integrated computational interface device. Insome embodiments, the keyboard may include a dedicated function key tobegin the wireless pairing process.

In some embodiments, the integrated computational interface device mayalso include at least one motion sensor located within the housing andat least one processor operatively connected thereto. The at least oneprocessor may be programmed to implement a mode of operation based oninput received from the at least one motion sensor. In some embodiments,the motion sensor may determine whether the integrated computationalinterface device is being moved and may adjust a mode of operation ofthe integrated computational interface device based on the movement. Insome embodiments, the motion sensor may determine whether the wearableextended reality appliance is moving relative to the integratedcomputational interface device or with the integrated computationalinterface device and may adjust a mode of operation of the integratedcomputational interface device or of the wearable extended realityappliance based on the movement. For example, if the user of thewearable extended reality appliance is walking, the number of itemsdisplayed to the user may be limited to prevent distracting the user.

The at least one motion sensor may include an accelerometer, agyroscope, a magnetometer, a motion sensor implemented using an imagesensor and by analyzing images captured using the image sensor with anegomotion algorithm, or other type of sensor configured to measuremotion of objects in the environment of the integrated computationalinterface device. For example, the at least one motion sensor may be themotion sensor 373 described above in connection with FIG. 3. The atleast one processor may include any processing device configured forreceiving input from the at least one motion sensor and configured to beprogrammed to implement a mode of operation based on the input.

In some embodiments, the at least one processor may be furtherprogrammed to automatically adjust settings of a virtual displaypresented by the wearable extended reality appliance based on the inputreceived from the at least one motion sensor. The at least one processormay include any processing device configured to be programmed toautomatically adjust one or more settings of the virtual displaypresented by the wearable extended reality appliance. The settings ofthe virtual display may be adjusted based on an environment where theuser is located (e.g., moving from indoors in low light to outdoors inbright light). The settings of the virtual display that may be adjustedmay include: picture settings, such as brightness, contrast, sharpness,or display mode (for example, a game mode with predefined settings);color settings, such as color component levels or other color adjustmentsettings; a position of the virtual display relative to a location ofthe user's head; or other settings that may enhance the user's view ofthe virtual display. In the embodiment shown in FIG. 1, the settings ofthe virtual screen 112 may be automatically adjusted by the at least oneprocessor.

In some embodiments, the at least one processor may be furtherprogrammed to output a notice when the integrated computationalinterface device is moved by more than a threshold distance while thewearable extended reality appliance is disengaged from the holder. Theat least one processor may include any processing device configured tobe programmed to output a notice when the integrated computationalinterface device is moved by more than a threshold distance while thewearable extended reality appliance is disengaged from the holder. Forexample, the notice may be provided to the user to alert the user thatthey may be moving the wearable extended reality appliance out ofwireless communication range, such that interaction with the integratedcomputational interface device will be interrupted unless the user movescloser than the threshold distance. As another example, if the wearableextended reality appliance is connected to the integrated computationalinterface device by a wire, the notice may be provided if the user isabout to move farther away from the integrated computational interfacedevice than the length of the wire, which may cause the wire to becomedisconnected, the wearable extended reality appliance to be accidentallyremoved from the user's head, or the integrated computational interfacedevice to be knocked off a surface where the integrated computationalinterface device is located.

The notice may include an alert, an alarm, or other audio and/or visualindicator. The notice may be output via any of the devices connected tooutput interface 350 shown in FIG. 3 (for example, light indicators 351,display 352, and/or speakers 353) or output interface 450 shown in FIG.4 (for example, light indicators 451, display 452, speakers 453, and/orprojector 454). The threshold distance may be a percentage of a lengthof a wire connected between the integrated computational interfacedevice and the wearable extended reality appliance, a fixed distancerelative to the length of the wire, a fixed distance from the integratedcomputational interface device, a percentage of a distance range of awireless communication protocol between the integrated computationalinterface device and the wearable extended reality appliance, a fixeddistance relative to the distance range of the wireless communicationprotocol, or any other distance that would take the wearable extendedreality appliance away from the integrated computational interfacedevice.

In some embodiments, the integrated computational interface device mayfurther include at least one sensor within the housing and at least oneprocessor operatively connected thereto. The at least one sensor may beconfigured to provide input indicative of whether the wearable extendedreality appliance is engaged with the holder. The at least one processormay be programmed to use the received input to implement a mode ofoperation based on whether the wearable extended reality appliance isengaged with the holder. For example, in response to an input indicativeof the wearable extended reality appliance being engaged with theholder, the at least one processor may implement a first mode ofoperation, and in response to an input indicative of the wearableextended reality appliance not being engaged with the holder, the atleast one processor may implement a second mode of operation, the secondmode of operation may differ from the first mode of operation. In someexamples, the at least one processor may be programmed to automaticallyadjust settings of a virtual display presented by the wearable extendedreality appliance based on whether the wearable extended realityappliance is engaged with the holder, for example as described hereinwith relation to the adjustment of the settings based on the input fromthe at least one motion sensor. In some examples, the at least oneprocessor may be programmed to output an audible indication when atleast one of the wearable extended reality appliance engages with theholder or the wearable extended reality appliance disengages from theholder occurs. In some examples, the mode of operation may be or mayinclude a power mode of at least one of: the at least one processor, acommunication device included in the integrated computational interfacedevice, or the wearable extended reality appliance. In one example, whenthe wearable extended reality appliance is engaged with the holder, thepower mode may be a turned-off mode, a sleep mode, a hibernation mode,and so forth. In another example, the power mode when the wearableextended reality appliance is engaged with the holder may be associatedwith lower power consumption than the power mode when the wearableextended reality appliance is not engaged with the holder (for example,using less hardware components when the wearable extended realityappliance is engaged with the holder, using lower clock-speed when thewearable extended reality appliance is engaged with the holder, and soforth).

In some examples, the mode of operation may include a display mode forpresentation of virtual content via the wearable extended realityappliance. In one example, in one mode of operation, no virtual contentmay be presented via the wearable extended reality appliance (forexample, when the wearable extended reality appliance is engaged withthe holder), while in another mode of operation, a selected virtualcontent may be presented via the wearable extended reality appliance(for example, when the wearable extended reality appliance is notengaged with the holder). In another example, in one mode of operation,virtual content may be presented via the wearable extended realityappliance at a smaller size (for example, when the wearable extendedreality appliance is engaged with the holder), while in another mode ofoperation the selected virtual content may be presented via the wearableextended reality appliance at a larger size (for example, when thewearable extended reality appliance is not engaged with the holder). Inanother example, in one mode of operation, virtual content may bepresented via the wearable extended reality appliance at a lower opacity(for example, when the wearable extended reality appliance is engagedwith the holder), while in another mode of operation the selectedvirtual content may be presented via the wearable extended realityappliance at a higher opacity (for example, when the wearable extendedreality appliance is not engaged with the holder). In another example,in one mode of operation, virtual content may be presented via thewearable extended reality appliance at a lower brightness (for example,when the wearable extended reality appliance is engaged with theholder), while in another mode of operation the selected virtual contentmay be presented via the wearable extended reality appliance at a higherbrightness (for example, when the wearable extended reality appliance isnot engaged with the holder).

In some examples, when the wearable extended reality appliance isengaged with the holder, the mode of operation may be selected based onvirtual content presented via the wearable extended reality appliancebefore the engagement of the wearable extended reality appliance withthe holder. For example, in response to a first virtual content (suchas, virtual content related hardware maintenance, virtual contentrelated to high priority tasks, etc.), a first mode of operation may beselected, and in response to a second virtual content (such as, virtualcontent requiring an involvement of a user, virtual content related tolow priority tasks, etc.), a second mode of operation may be selected,the second mode may differ from the first mode. In some examples, whenthe wearable extended reality appliance is engaged with the holder, themode of operation may be selected based on an analysis of image datacaptured using at least one image sensor (for example, at least oneimage sensor included in the wearable extended reality appliance,included in the integrated computational interface device, and soforth). For example, the image data may be analyzed using a visualclassification algorithm to classify the physical environment of theintegrated computational interface to a particular class out of aplurality of alternative classes, and the mode of operation may beselected based on the particular class. Some non-limiting examples ofsuch classes may include “outdoor,” “indoor,” “office,” “home,” “meetingroom,” “at least one person in the environment,” “at least two personsin the environment,” “no person in the environment,” and so forth. Inanother example, the image data may be analyzed using a visual motionrecognition algorithm to detect motion in the physical environment ofthe integrated computational interface, and the mode of operation may beselected based on whether motion towards the integrated computationalinterface is identified.

In some embodiments, the integrated computational interface device mayalso include a protective cover. The cover may protect a portion of thehousing from damage during transportation, for example, a portion of thehousing including the key region, the non-key region, and the holder.The protective cover may be completely removable from the housing or maybe attached by one or more sides thereof to the housing. One side of theprotective cover may be fixedly attached to the housing. The protectivecover may include two layers of a soft material (such as a non-woventextile) encasing a second material (such as silicon). The protectivecover may include a first layer of a soft material (such as a non-woventextile) and a second layer of a second material (such as silicon). Theprotective cover may be made of any number of layers or of differenttypes of materials to provide bump, shock, or impact protection to thekeyboard and/or the housing and/or the wearable extended realityappliance when the wearable extended reality appliance is selectivelyengaged with the holder.

The protective cover may be operable in two encasing modes. In a firstencasing mode, the protective cover may be configured to overlie thewearable extended reality appliance in the housing. For example, in thefirst encasing mode, the protective cover may provide bump, shock, orimpact protection to the keyboard and/or the housing and/or the wearableextended reality appliance when the wearable extended reality applianceis selectively engaged with the holder. The protective cover may includeone or more features, such as one or more protrusions, configured tohold the wearable extended reality appliance in the first encasing mode.For example, the protective cover may include one or more fold lines toallow the protective cover to bend over the wearable extended realityappliance when the wearable extended reality appliance is selectivelyengaged with the holder, and to bend differently when the wearableextended reality appliance is selectively disengaged from the holder.

In a second encasing mode, the protective cover may be configured toelevate the housing. For example, in the second encasing mode, theprotective cover may allow the housing to be elevated relative to asurface where the housing is placed, such as a table, a desk, or auser's lap. To elevate the housing, the protective cover may besegmented such that the protective cover may be folded into differentpositions whereby the housing may be elevated to one or more distancesabove the surface on which the housing is placed. In the second encasingmode, the protective cover may not overlie the wearable extended realityappliance.

FIG. 13A is a right side perspective view of integrated computationalinterface device 1310 having protective cover 1312 in a first encasingmode. FIG. 13A is an exemplary representation of just one embodiment,and it is to be understood that some illustrated elements might beomitted and others added within the scope of this disclosure. Protectivecover 1312 may cover an upper portion of housing 1314 while a bottomsurface of housing 1314 contacts surface 1316 where integratedcomputational interface device 1310 may be placed. In some non-limitingexamples, integrated computational interface device 1310 may include allor some of the elements of at least one of integrated computationalinterface device 610, integrated computational interface device 710, orintegrated computational interface device 1210.

FIG. 13B is a left side perspective view of integrated computationalinterface device 1310 of FIG. 13A having protective cover 1312 in asecond encasing mode. FIG. 13B is an exemplary representation of justone embodiment, and it is to be understood that some illustratedelements might be omitted and others added within the scope of thisdisclosure. When protective cover 1312 is in the second encasing mode,protective cover 1312 may be at least partially separated from housing1314 to permit access to keyboard 1318, wearable extended realityappliance 1320, and holder 1322 for wearable extended reality appliance1320. Protective cover 1312 may be segmented to create individualsegments (such as segments 1324, 1326, 1328, and 1330) such thatprotective cover 1312 may be folded to elevate at least a portion ofhousing 1314 above surface 1316. The housing may be elevated by placingthe folded protective cover 1312 under one end of the housing to elevatethat end of the housing. It is noted that the number of segments1324-1330 shown in FIG. 13B is exemplary and that protective cover 1312may have fewer segments or more segments.

The protective cover may also include at least one camera associatedtherewith. For example, the protective cover may include one or morecameras or other types of imaging devices configured to capture imagesof the keyboard, the user, or the environment around the user. The oneor more cameras may include one or more selfie cameras, back cameras, orother cameras associated with the protective cover such that the cameramay be usable when the protective cover is in the second encasing mode.

The protective cover may also include at least one protrusion on atleast two sides of the at least one camera. For example, the protectivecover may include two protrusions from two sides of the at least onecamera, one protrusion encircling the at least one camera from at leasttwo sides, or another number of protrusions such that the at least oneprotrusion is configured to prevent the at least one camera fromcontacting a flat surface when the cover is positioned on the flatsurface with the camera facing the flat surface.

FIG. 14 is a right side perspective view of another embodiment ofintegrated computational interface device 1410 having protective cover1412 in a second encasing mode. FIG. 14 is an exemplary representationof just one embodiment, and it is to be understood that some illustratedelements might be omitted and others added within the scope of thisdisclosure.

When protective cover 1412 is in the second encasing mode, protectivecover 1412 is at least partially separated from housing 1414 to permitaccess to keyboard 1416, wearable extended reality appliance 1418, andholder 1420 for wearable extended reality appliance 1418. Protectivecover 1412 may be segmented to create individual segments (such assegments 1422, 1424, and 1426) such that protective cover 1412 may befolded to elevate at least a portion of housing 1414 above surface 1428.It is noted that the number of segments 1422-1426 shown in FIG. 14 isexemplary and that protective cover 1412 may have fewer segments or moresegments.

Camera 1430 may be positioned in protective cover 1412 in segment 1422.It is noted that camera 1430 may be positioned in any portion ofprotective cover 1412. Three protrusions 1432 extend outward fromprotective cover 1412 to form a triangular shape around camera 1430.Protrusions 1432 are configured to prevent camera 1430 from contactingsurface 1428 when protective cover 1412 is positioned on surface 1428with camera 1430 facing surface 1428.

Traditionally, cameras are located on a top frame of a laptop screen.This position enables the camera to face the user when the laptop isused from an elevated position in relation to the surface that thelaptop is placed on. This elevated position allows for a desirableviewing angle of the user. Some disclosed embodiments allow for a laptopwithout a significant physical screen using an extended realityappliance. However, this raises the problem of camera position.Positioning the camera on the keyboard creates an undesirable viewingangle of the user that users are not accustomed to. Therefore, it isdesirable to position the camera in an elevated position in relation tothe keyboard. One solution provided by some disclosed embodimentsincludes locating the camera in a foldable cover of the keyboard,configured to be folded to a configuration in which the camera iselevated above the keyboard and faces the user.

In some embodiments, an integrated computational interface device mayinclude a housing having a key region and a non-key region. Anintegrated computational interface may include any device that has anumber of functional components. The device may serve as an interface inpermitting human interaction with a machine such as a computer. In oneexample, an integrated computational interface device may include acomputing device configured to work with a wearable extended realityappliance external to the integrated computational interface device, forexample to enable a presentation of an extended reality environment viathe wearable extended reality appliance. In one example, an integratedcomputational interface device may include, for example in a singlehousing, a computing device (such as a processor, a CPU, etc.),integrated with an input device (such as a keyboard, a touchscreen, atouchpad, etc.), a digital communication device configured to connectthe integrated computational interface device to a digital communicationnetwork (such as Ethernet, cellular network, the Internet, etc.), and acommunication port configured to connect a wearable extended realityappliance external to the integrated computational interface device.

A housing may include any physical structure in which one or morecomponents are contained or housed. Such a housing may have a keyregion, such as a general area where keys are located and a non-keyregion, generally devoid of keys. A key may include any button, switch,trigger, toggle, or any other element capable of activation via physicalmanipulation. In some embodiments, keys may be mechanical in nature(e.g., mechanical push button(s) such as found on a typical computerkeyboard). In other instances, keys may be soft (e.g., a touch displayon which images of key(s) are simulated). By way of example, a keyregion may cover an area of an alphanumeric keyboard or a numerickeyboard, while a non-key region may be another area of the interfacelacking keys. In some examples, the key region and/or the non-key regionmay be regions of an external surface (or of an upper external surface)of the integrated computational interface device. In some embodiments,an integrated computational interface device may include a housinghaving an input region and a non-input region. In one example, the inputregion may be an external surface area (or an upper external surfacearea) of the integrated computational interface device including atouchscreen. In another example, the input region may be an externalsurface area (or an upper external surface area) of the integratedcomputational interface device including a touchpad. In yet anotherexample, the input region may be and/or include a key region asdescribed above. In one example, the non-input region may be an externalsurface area (or an upper external surface area) of the interfacelacking any input device. In another example, the input region may be anexternal surface area (or an upper external surface area) of theintegrated computational interface device including an input device of aparticular type, and the non-input region may be another externalsurface area (or another top external surface area) of the integratedcomputational interface device lacking input devices of the particulartype.

A housing may include an outer covering or shell. The housing maysurround components of the integrated computational interface device andmay cover some or all components of the integrated computationalinterface device. It is contemplated that the housing may have one ormore openings that may expose certain components (e.g., USB or otherports or an image sensor) of the integrated computational interfacedevice or may allow certain components (e.g., keys of a keyboard) toprotrude from the housing. The housing may support certain components ofthe integrated computational interface device (e.g., a circuit board) inan interior portion of the housing.

The housing of the integrated computational device may include a keyregion and a non-key region distinct from the key region. As previouslydiscussed in greater detail, the key region may include one or more keysthat may allow a user to enter alphanumeric or other characters asinputs. For example, in some embodiments, a keyboard may be associatedwith the key region of the housing. The keyboard may be a standardtypewriter-style keyboard (e.g., a QWERTY-style keyboard) or othersuitable keyboard layout, such as a Dvorak layout or a chorded layout.The keyboard may include any suitable number of keys; for example, a“full size” keyboard may include up to 104 or 105 keys. In one example,the keyboard may include at least 10 keys, at least 30 keys, at least 80keys, at least 100 keys, and so forth. In one example, the keyboard maybe included in the integrated computational interface device, in anexternal surface area of the integrated computational interface device,in an upper external surface area of the integrated computationalinterface device, in the housing, in an external surface of the housing,in an upper external surface of the housing, in the key region, and soforth. In some embodiments, an input device may be associated with theinput region of the housing. Some non-limiting examples of such an inputdevice may include a touchscreen, a touchpad, a keyboard as describedabove, and so forth. For example, the input device may be a touchscreen,and the touchscreen may have a diagonal length of at least 1 inch, atleast 5 inches, at least 10 inches, and so forth. In another example,the input device may be a touchpad, and the touchpad may have a diagonallength of at least 1 inch, at least 5 inches, at least 10 inches, and soforth. In one example, the input device may be included in theintegrated computational interface device, in an external surface areaof the integrated computational interface device, in the housing, in anexternal surface of the housing, in an upper external surface area ofthe integrated computational interface device, in the housing, in anupper external surface of the housing, in the key region, and so forth.

As previously discussed in greater detail, the non-key region may be aregion of the housing that does not include any keys and may exist tocomplete a desired shape of the housing that extends beyond the keyregion of the housing in any direction. The non-key region may be anarea that may include an input element, such as a track pad, a trackball, a touchscreen, a touch bar, or other form of cursor control forthe integrated computational interface device. The non-key region may besubdivided into multiple different non-key regions, such as the trackpad or other cursor control, an extension of the housing, or a cover orgrille for one or more speakers or other audio output devices includedwithin the housing. In some embodiments, the housing may include two ormore non-key regions. For example, a first non-key region may bepositioned at a top edge of the key region and a second non-key regionmay be positioned at a bottom edge of the key region. In someembodiments, the non-key region may just be a portion of the housing,without any function other than to serve as a portion of the housing. Insome examples, as previously discussed in greater detail, the non-inputregion may be a region of the housing that does not include any inputdevice, or that does not include any input device of a particular type,and may exist to complete a desired shape of the housing that extendsbeyond the input region of the housing in any direction. In someembodiments, the housing may include two or more non-input regions.

In some embodiments, the integrated computational interface device mayinclude at least one image sensor. An image sensor may include a devicethat converts photons (i.e., light) into electrical signals forinterpretation. For example, an image sensor may incorporate acharge-coupled device (CCD) or an active-pixel sensor (CMOS sensor),fabricated in complementary MOS (CMOS) or N-type MOS (NMOS or Live MOS)technologies. The at least one image sensor may be configured to captureimages and/or videos of a user of the integrated computational interfacedevice or may otherwise be located in a physical environment of theuser. As previously described, an image sensor may be configured tocapture visual information by converting light to image data. In someembodiments, the at least one image sensor may be at least one of: acolor image sensor, a monochromatic image sensor, a stereo image sensor,an infrared image sensor, or a depth image sensor.

In some examples, image data captured using the at least one imagesensor included in the integrated computational interface device may beanalyzed to determine whether a user of a wearable extended realitydevice is approaching the integrated computational interface device. Forexample, the image data may be analyzed using a visual classificationalgorithm to determine whether a user of a wearable extended realitydevice is approaching the integrated computational interface device,whether a person approaching the integrated computational interfacedevice is a user of a wearable extended reality device or not, and soforth. In some examples, the image data may be analyzed to identify thewearable extended reality device used by a person approaching theintegrated computational interface device. For example, a unique visualcode may be presented on the wearable extended reality device (forexample, in a sticker or on a display screen on an external side of thewearable extended reality device), the image data may be analyzed todetect and recognize the unique visual code, and a data structureassociating wearable extended reality devices with visual codes may beaccessed based on the identified visual code to identify the wearableextended reality device. Further, in some examples, pairing of theintegrated computational interface device with the identified wearableextended reality device may be initiated upon the identification of thewearable extended reality device.

In some embodiments, the integrated computational interface device mayinclude a foldable protective cover incorporating the at least one imagesensor wherein the protective cover may be configured to be manipulatedinto a plurality of folding configurations. A cover may be consideredfoldable if it has a non-rigid structure, enabling it to at leastsomewhat adjust or conform to a structure positioned beneath the cover.By way of example, a foldable cover may have distinct fold lines orcreases, or may be generally flexible without specific creases.Foldability may be facilitated through creases in the cover, and/or maybe facilitated based on its material composition. For example, the covermay be made of a flexible material that enables it to fold.

In some folding configurations, the cover may protect a portion of thehousing from damage during transportation, such as a portion of thehousing including the key region and at least a portion of the non-keyregion (or the input region and at least a portion of the non-inputregion). In some embodiments, if the housing has multiple non-keyregions, the cover may not cover all non-key regions. For example, insome embodiments, the cover may not extend over a non-key regionpositioned at a bottom edge of the key region or may cover only aportion of the non-key region positioned at the bottom edge of the keyregion. In some embodiments, if the housing has multiple non-inputregions, the cover may not cover all non-input regions. For example, insome embodiments, the cover may not extend over a non-input regionpositioned at a bottom edge of the input region or may cover only aportion of the non-input region positioned at the bottom edge of theinput region.

The cover may incorporate an image sensor. For example, an aperture inthe cover may expose a lens of the image sensor, the image sensor may beembedded within layers of the cover (with an aperture in at least onelayer for exposing a lens of the image sensor), the image sensor may beaffixed to an external surface of the cover, or the image sensor may beconfigured for connection or association with the cover in any otherway.

In some embodiments, the protective cover may have a quadrilateral shapewith one side connected to the housing. In some embodiments, the covershape may not be perfectly quadrilateral. For example, the cornersand/or edges of the cover may be rounded, beveled, or chamfered as aresult of its fabrication process. In some embodiments, the protectivecover may have a similar shape as a top surface of the housing such thatthe protective cover completely covers the top surface of the housing,including covering the key region and the at least a portion of thenon-key region (or including covering the input region and the at leasta portion of the non-input region). In some embodiments, the cover maybe configured to extend beyond one or more edges of the top surface ofthe housing such that the cover may at least partially wrap around oneor more sides of the housing.

In some embodiments, the at least one image sensor may be located closerto a first side of the protective cover connected to the housing than toa second side of the protective cover, opposite the first side. In someembodiments, the first side of the protective cover may be connected tothe housing by being fixedly attached to the housing. For example, thefirst side of the protective cover may be attached or coupled to thehousing by a hinge mechanism. A rotating hinge may be employed, or thehinge might be made of a flexible plastic or fabric material. These arejust a few examples.

Any type of hinge mechanism known in the art may be used. Although not arequirement, in some embodiments, the hinge mechanism may be made of aflexible material, with a first edge of the flexible material fixedlyattached to the housing and a second edge, opposite the first edge,fixedly attached to the protective cover. In some embodiments, theflexible material may include fabric, silicone, rubber, or otherpolymeric or elastomeric material. In some embodiments, the hingemechanism may include one or more knuckles (e.g., rigid or semi-rigidloop structures) connected to and extending away from the protectivecover, one or more knuckles connected to and extending away from thehousing, and a pin inserted through all of the knuckles to create thehinge, such that the protective cover may rotate or pivot around thepin. When assembled together prior to inserting the pin, the knuckles ofthe protective cover and the knuckles of the housing may be aligned orinterleaved.

In some embodiments, the first side of the protective cover may beremovably connected to the housing. For example, the first side of theprotective cover may include one or more magnets and the housing mayinclude a ferromagnetic material to enable interaction with the magnetssuch that the protective cover may be removably connected to thehousing. As another example, the first side of the protective cover mayinclude a ferromagnetic material and the housing may include one or moremagnets such that the protective cover may be removably connected to thehousing.

In some embodiments, the protective cover may include multiple (e.g.,two) layers of a relatively soft material (such as a polymeric sheet,fabric, non-woven textile, or similar soft material) encasing a secondmaterial (such as silicone). In some embodiments, the protective covermay include a first layer of a first material (for example, a non-woventextile) and a second layer of a second material (for example,silicone). In some embodiments, the protective cover may be made of anynumber of layers or of different types of materials to provide bump,shock, or impact protection to the keyboard and/or the input deviceand/or the housing.

In some embodiments, electronics of the at least one image sensor may besandwiched between a first external layer of the protective cover and asecond external layer of the protective cover. The sandwiching may occurthrough stitching, gluing, bonding, or any other form of fixationbetween at least some portions of adjacent layers. In some embodiments,the at least one image sensor may be mounted in an image sensor housingto protect the at least one image sensor from bumps, shocks, orvibrations. Electronics of the at least one image sensor may support oneor more operations of the at least one image sensor. For example, theelectronics may include some or substantially all the components of theat least one image sensor that enables it to function in an intendedmanner. In some embodiments, the electronics may include, for example, apower connection to supply power to the at least one image sensor and adata connection to transfer data from the at least one image sensor toother components of the integrated computational interface device. Insome embodiments, the power connection and the data connection may beprovided by a single wire. In some embodiments, the power connection maybe provided by a separate wire from the data connection. In someembodiments, the image sensor housing may be configured to hold oneimage sensor or multiple image sensors. In embodiments with multipleimage sensors, each image sensor may have a separate power connectionand data connection or may share a power connection and/or a dataconnection via a bus connection, with each image sensor connected to thebus to share the power connection and/or the data connection. In someembodiments, the electronics may be contained in the image sensorhousing.

In some embodiments, when the protective cover is in a first foldingconfiguration, the first external layer of the protective cover may facethe housing and the second external layer of the protective cover may beon an opposite side of the protective cover, such that the secondexternal layer faces away from the housing. In some embodiments, the atleast one image sensor may be located between the first external layerand the second external layer, to “sandwich” the at least one imagesensor between the first external layer and the second external layer tosupport the at least one image sensor and to provide protection to theat least one image sensor against bumps, shocks, or vibrations. In sucha configuration, the first external layer may include an opening suchthat the at least one image sensor is not covered by the first externallayer and that the at least one image sensor may capture images. In someembodiments, the image sensor housing may be located between the firstexternal layer and the second external layer.

In some embodiments, each of the first external layer and the secondexternal layer may be made from a single continuous material and whereinthe electronics may be located on an intermediate layer made from aplurality of separated elements. In some embodiments, the first externallayer and the second external layer may each made of a single piece ofmaterial and may be connected to each other around the respective edgesthereof, for example, by stitching, compression, melting, an adhesive,or other methods of fusing the edges together such that the plurality ofseparated elements of the intermediate layer are contained between thefirst external layer and the second external layer.

In some embodiments, the plurality of separated elements may besurrounded on opposite sides by the first external layer and the secondexternal layer, wherein the plurality of separated elements issandwiched between the first external layer and the second externallayer. In some embodiments, the plurality of separated elements may bepositioned in “pockets” formed between the first external layer and thesecond external layer, such that each separated element is in a separatepocket. In some embodiments, the plurality of separated elements maypermit the protective cover to be flexible and formed in a plurality ofconfigurations, including a flat configuration and one or morethree-dimensional configurations. Additional details about possibleshapes of the protective cover will be described below.

In some embodiments, the electronics of the at least one image sensormay be attached to one of the plurality of separated elements. Forexample, the electronics may be attached to one of the plurality ofseparated elements by an adhesive. Any suitable type of adhesive may beused. As another example, the electronics may be positioned in a firstsupport element and one of the plurality of separated elements mayinclude a second support element, such that the first support elementand the second support element interact (e.g., by a tongue and groovearrangement, a slide-fit arrangement, a snap-fit arrangement, or apress-fit arrangement) to attach the first support element to the secondsupport element. In some embodiments, the first support element mayinclude the image sensor housing previously described and the secondsupport element may include a bracket configured to interact with theimage sensor housing to attach the image sensor housing to one of theplurality of separated elements. In some embodiments, the electronicsmay be permanently attached to one of the plurality of separatedelements. In some embodiments, the electronics may be removably attachedto one of the plurality of separated elements. In some embodiments, theelectronics may be integrally formed with one of the plurality ofseparated elements. For example, one of the plurality of separatedelements may be molded or formed around the electronics to contain theelectronics therein.

In some embodiments, at least some of the plurality of separatedelements may have triangular shapes. In some embodiments, at least someof the plurality of separated elements may have quadrilateral shapes. Insome embodiments, the quadrilateral shaped elements may have rounded,chamfered, or beveled edges and/or corners. In some embodiments, atleast some of the plurality of separated elements may have other shapes,such as circular, semi-circular, elliptical, oval, or another shape. Ingeneral, the separated elements may have any shape with linear ornon-linear (e.g., curved) sides. In some embodiments, all of theplurality of separated elements may have substantially the same shape.In some embodiments, some of the plurality of separated elements mayhave a first shape and other separated elements may have a second shapedifferent from the first shape. In some embodiments, at least some ofthe plurality of separated elements may have substantially the sameshape in different sizes. For example, some of the plurality ofseparated elements may have substantially triangular shapes of differentsizes; e.g., small triangular shapes and large triangular shapes. Thedescribed shapes for the plurality of separated elements are approximatein that each of the elements may have rounded, chamfered, or bevelededges and/or corners.

In some embodiments, the first external layer and the second externallayer may be made from a first material and the intermediate layer mayinclude a second material that differs from the first material. In someembodiments, the first external layer may be made of a differentmaterial than the second external layer. In some embodiments, the secondmaterial of the intermediate layer may be a different material thaneither the first external layer or the second external layer.

In some embodiments, the first material and the second material mayinclude plastic materials, silicon, carbon-fiber, polycarbonate, fabric,or leather. For example, the first material may be a polycarbonatematerial and the second material may be silicon or fabric. As anotherexample, the first material may be a first type of fabric and the secondmaterial may be a second type of fabric. As another example, the secondmaterial may be at least one of glass, ceramic, epoxy, plastic, orpolytetrafluoroethylene. In some embodiments, the first material may bemore durable than the second material. In some embodiments, the secondmaterial may be softer than the first material. In some embodiments, thesecond material may be a dielectric material.

In some embodiments, the first material may be more rigid than thesecond material. In some embodiments, the first material may be rigid orsemi-rigid. In some embodiments, the second material may be more rigidthan the first material. In some embodiments, the second material may berigid or semi-rigid.

In some embodiments, in a first folding configuration, the protectivecover may be configured to encase the key region and at least a portionof the non-key region. In some embodiments, in the first foldingconfiguration, the protective cover may encase an entire upper surfaceof the housing, including the key region and the entire non-key region.For example, if the protective cover may be connected to the housing byone side, and in the first folding configuration, the protective covermay extend from a side of the housing where the protective cover isconnected to an opposite side of the housing away from where theprotective cover is connected. In another example, in the first foldingconfiguration, the protective cover may encase a portion of the uppersurface of the housing, including the key region and at least a portionof the non-key region. In some embodiments, in the first foldingconfiguration, the protective cover may be configured to encase theinput region and at least a portion of the non-input region. Forexample, the protective cover may encase a portion of the upper surfaceof the housing, including the input region and at least a portion of thenon-input region. In some embodiments, the protective cover may lay flatacross the upper surface of the housing.

In some embodiments, the protective cover may be removably connected tothe housing at the opposite side of the housing, for example, by amagnetic connection. In some examples, the integrated computationalinterface device may include a sensor, and data captured using thesensor may be indicative of whether the protective cover is in the firstfolding configuration. Some non-limiting examples of such a sensor mayinclude a proximity sensor, a magnetic sensor, a physical button(functioning as a switch), and so forth. Further, in some examples, theintegrated computational interface device may be configured to withholdimage and/or video capturing using the at least one image sensor whenthe protective cover is in the first folding configuration. In otherexamples, the integrated computational interface device may beconfigured to change operational power mode (such as sleep, hibernate,regular, etc.) based on whether the protective cover is in the firstfolding configuration.

In some embodiments, in a second folding configuration, the protectivecover may be configured to stand in a manner that causes an optical axisof the at least one image sensor to generally face a user of theintegrated computational interface device while the user types on thekeyboard. For example, when the cover is folded to a secondconfiguration where at least a portion of the cover no longer covers theinterface device, at least that portion of the cover may be configuredto assume an elevated position where the image sensor faces in adirection of the user while the user interacts with the interface device(e.g., uses the device, for example, types on the keyboard). In someembodiments, the protective cover may be folded into one or moresectors, with at least some sectors assisting in supporting theprotective cover in the second folding configuration. In someembodiments, when in the second folding configuration, the protectivecover may be folded into a substantially triangular shape. The opticalaxis of the at least one image sensor may be a virtual line through thecenter of a lens of the image sensor and may help to define an angle ofview of the image sensor such that the image sensor may capture an imageof the user, for example while the user types on the keyboard. In someembodiments, in the second folding configuration, the protective covermay be configured to stand in a manner that causes an optical axis ofthe at least one image sensor to generally face a user of the integratedcomputational interface device while the user physically interacts withthe input device. One non-limiting example of such physical interactionwith the input device may include touching the input device (forexample, touching the touchscreen, touching the touchpad, and so forth).

In some embodiments, in the second folding configuration, when thehousing rests on a surface, a sector of the protective cover opposingthe side of the quadrilateral shape connected to the housing may also beconfigured to rest on the surface. In some embodiments, the sector ofthe protective cover that rests on the surface helps the protectivecover to stand in the second folding configuration. In some embodiments,the sector of the protective cover extends away from the housing whileresting on the surface. In some embodiments, the sector of theprotective cover extends toward the housing while resting on thesurface.

In some embodiments, an area of the sector of the protective coverconfigured to rest on the surface in the second folding configurationmay be at least 10% of an overall area of the protective cover. In someembodiments, the area of the sector of the protective cover that restson the surface in the second folding configuration may be larger orsmaller than 10% of the overall area of the protective cover, such thatthe area of the protective cover that rests on the surface assists theprotective cover in staying in the second folding configuration.

In some embodiments, a side of the sector of the protective cover thatrests on the surface may include a non-slip portion such that the sectordoes not slide when resting on the surface. For example, the non-slipportion may include a textured portion, such as ribbed, grooved, orpebble grained. As another example, the non-slip portion may be made ofa non-slip material, such as a hard plastic or rubber. In someembodiments, a size of the non-slip portion is less than the area of thesector of the protective cover that rests on the surface.

In some embodiments, the protective cover may include flexible sectionsthat enable folding the protective cover along a plurality of predefinedfolding lines. Such folding lines may, for example, be creases in thecover. In some embodiments, the flexible sections may be between one ormore of the plurality of separated elements of the intermediate layer ofthe protective cover. In some embodiments, the flexible sections may bebetween each of the plurality of separated elements of the intermediatelayer of the protective cover. In some embodiments, the flexiblesections may be formed in locations where the first external layer andthe second external layer of the protective cover meet. In someembodiments, the predefined folding lines may correspond to the flexiblesections of the protective cover. For example, any flexible section ofthe protective cover may be a predefined folding line. In someembodiments, the flexible sections may permit the protective cover to befolded in a plurality of shapes, including one or more three-dimensionalshapes. In some embodiments, a predefined folding line may be a portionof the protective cover along which the protective cover may be folded.

In some embodiments, at least some of the folding lines may benon-parallel to each other and the flexible sections may enable foldingthe protective cover to form a three-dimensional shape that includes acompartment for selectively enclosing a wearable extended realityappliance. In some embodiments, the wearable extended reality appliancemay include a pair of smart glasses or goggles, as previously described.Smart glasses may appear similar to traditional eyeglasses and mayinclude “smart” functionality such as a camera positioned to takepictures of what the user is presently viewing or one or more displaysconfigured to project images onto lenses of the smart glasses. In someembodiments, wearable extended reality appliance may take other formsthat may include one or more lenses, such as goggles or other form ofwearable device.

In some embodiments, the wearable extended reality appliance may beplaced on the non-key region and/or the key region of the housing, andin the first folding configuration, the protective cover may enclose thewearable extended reality appliance while placed on the non-key regionand/or the key region. The compartment formed in the protective covermay provide room between an inner surface of the protective cover and asurface of the housing including the non-key region and the key regionsuch that the wearable extended reality appliance may be enclosedtherein.

In some examples, the integrated computational interface device mayfurther include at least one projector, and the foldable protectivecover may further incorporate the at least one projector. For example,the at least one projector may be configured to emit at least one ofvisible light, infrared light, or near infrared light. In one example,the optical axis of the at least one image sensor may be parallel to acentral direction of the at least one projector. In one example, anactive stereo camera may be implemented using the at least one projectorand the at least one image sensor. In another example, a time-of-flightcamera may be implemented using the at least one projector and the atleast one image sensor. In some examples, the at least one projector maybe activated when the protective cover is in the second foldingconfiguration and may be deactivated when the protective cover is in thefirst folding configuration and/or is in the third foldingconfiguration. In some examples, the at least one projector may beactivated when the protective cover is in the third foldingconfiguration and may be deactivated when the protective cover is in thefirst folding configuration and/or is in the second foldingconfiguration. In some examples, the at least one projector may includea first projector positioned at a first distance from the at least oneimage sensor and a second projector positioned at a second distance fromthe at least one image sensor, the second distance may be larger thanthe first distance. In one example, in the second folding configuration,the first projector may be activated and the second projector may bedeactivated, and in the third folding configuration, the secondprojector may be activated and the first projector may be deactivated.

In some embodiments, the plurality of predefined folding lines mayinclude at least two traversing folding lines and at least twonon-traversing folding lines. In some embodiments, a traversing foldingline may be a folding line that crosses one or more other folding lines.In some embodiments, a non-traversing folding line may be a folding linethat does not cross another folding line (e.g., non-traversing foldinglines may be parallel to each other). In some embodiments, thetraversing folding lines and the non-traversing folding lines may enablethe protective cover to be folded into the first folding configuration,into the second folding configuration, and into one or morethree-dimensional shapes.

In some examples, in a third folding configuration, the protective covermay be configured to be in a position that causes an optical axis of theat least one image sensor to generally face away from the user. Forexample, in the third folding configuration, the cover may be adjacentto a rear surface of the housing of the integrated computationalinterface device (for example, opposite to the keyboard). In someexamples, the integrated computational interface device may include asensor, and data captured using the sensor may be indicative of whetherthe protective cover is in the third folding configuration. Somenon-limiting examples of such sensor may include a proximity sensor, amagnetic sensor, a physical button (functioning as a switch), and soforth. In some examples, the integrated computational interface devicemay adjust at least one image capturing parameter of the at least oneimage sensor based on whether the protective cover is in the thirdfolding configuration. For example, when the protective cover is in thethird folding configuration, the at least one image capturing parameter(such as focus, field of view, resolution, and so forth) may be adjustedfor capturing images of farther objects. In one example, the at leastone image capturing parameter may include at least one of field of view,zoom, resolution, focus, frame rate, or color correction.

FIG. 15 is a front perspective view of a first exemplary embodiment ofan integrated computational interface device 1500. FIG. 15 is anexemplary representation of just one embodiment, and it is to beunderstood that some illustrated elements might be omitted and othersadded within the scope of this disclosure. In some examples,computational interface device 1500 may include elements ofcomputational interface devices 1600, 1700, 1800, 2000, 2100, and 2200that are not described herein in relation to computational interfacedevice 1500 and/or that are not shown in FIG. 15.

Integrated computational interface device 1500 includes a housing 1502having a key region 1504 and a non-key region 1506. A keyboard 1508 maybe associated with key region 1504. An image sensor 1510 may be includedin integrated computational interface device 1500. It is noted thatwhile FIG. 15 shows one image sensor 1510, additional image sensors maybe provided. In some embodiments, image sensor 1510 may be included inan image sensor housing 1512.

Integrated computational interface device 1500 may include a foldableprotective cover 1514. Foldable protective cover 1514 may incorporateimage sensor 1510 and/or image sensor housing 1512. As shown in FIG. 15,foldable protective cover 1514 is in a second folding configuration,such that foldable protective cover 1514 is configured to stand and thatimage sensor 1510 may generally face a user of integrated computationalinterface device 1500 while the user types on keyboard 1508.

Foldable protective cover 1514 may be connected to housing 1502 by aconnector 1516. As described herein, connector 1516 may include a fixedconnection (e.g., a hinge mechanism) or a detachable connection (e.g., amagnetic connection).

Foldable protective cover 1514 may include multiple sectors. As shown inFIG. 15, foldable protective cover 1514 may include a first sector 1518,a second sector 1520, and a third sector 1522. Sectors 1518, 1520, and1522 may enable foldable protective cover 1514 to stand in the secondfolding configuration. First sector 1518 is shown in FIG. 15 as restingon a surface where housing 1502 is resting and being folded under aportion of housing 1502. Third sector 1522 may be connected to housing1502 by connector 1516. In this example, image sensor 1510 and/or imagesensor housing 1512 may be included in sector 1522.

FIG. 16 is a front perspective view of a second exemplary embodiment ofan integrated computational interface device 1600. FIG. 16 is anexemplary representation of just one embodiment, and it is to beunderstood that some illustrated elements might be omitted and othersadded within the scope of this disclosure. For ease of description,elements shown in FIG. 16 that are similar to elements shown in FIG. 15are referenced with similar reference numerals. For example, integratedcomputational interface device 1600 may include key region 1604 similarto key region 1504, non-key region 1606 similar to non-key region 1506,wire port 1624 similar to wire port 1524, and so forth. In one example,computational interface device 1600 may be linked with wire 1626 similarto wire 1526, wearable extended reality appliance 1628 similar towearable extended reality appliance 1528, and so forth. In someexamples, computational interface device 1600 may include elements ofcomputational interface devices 1500, 1700, 1800, 2000, 2100, and 2200that are not described herein in relation to computational interfacedevice 1600 and/or that are not shown in FIG. 16.

Integrated computational interface device 1600 may include a foldableprotective cover 1614. Foldable protective cover 1614 may incorporateimage sensor 1610. As shown in FIG. 16, foldable protective cover 1614is in a second folding configuration, such that foldable protectivecover 1614 is configured to stand and that image sensor 1610 maygenerally face a user of integrated computational interface device 1600while the user types on keyboard 1608. Foldable protective cover 1614may be connected to housing 1602 by a connector 1616.

Foldable protective cover 1614 may include multiple sectors. As shown inFIG. 16, foldable protective cover 1614 may include a first sector 1618,a second sector 1620, and a third sector 1622. Sectors 1618, 1620, and1622 may enable foldable protective cover 1614 to stand in the secondfolding configuration. First sector 1618 is shown in FIG. 16 as restingon a surface where housing 1602 is resting and being folded away fromhousing 1602. Third sector 1622 may be connected to housing 1602 byconnector 1616.

Foldable protective cover 1614 may include predefined folding lines1630. A predefined folding line may be any line predisposed, duringmanufacture, for folding. Predefined folding lines 1630 may separatefoldable protective cover 1614 into sectors 1618, 1620, and 1622. Somepredefined folding lines 1630 may extend across a full distance offoldable protective cover 1614 and may be parallel to one another. Otherpredefined folding lines 1630 may extend in different directions. Apattern may be created by predefined folding lines 1630 to enablefoldable protective cover 1614 to be folded into one or morethree-dimensional shapes.

FIG. 17 is a top view of a third exemplary embodiment of an integratedcomputational interface device 1700. FIG. 17 is an exemplaryrepresentation of just one embodiment, and it is to be understood thatsome illustrated elements might be omitted and others added within thescope of this disclosure. For ease of description, elements shown inFIG. 17 that are similar to elements shown in FIG. 15 are referencedwith similar reference numerals. For example, integrated computationalinterface device 1700 may include key region 1704 similar to key region1504, non-key region 1706 similar to non-key region 1506, keyboard 1708similar to keyboard 1508, wire port 1724 similar to wire port 1524, andso forth. In one example, computational interface device 1700 may belinked with wire 1726 similar to wire 1526, wearable extended realityappliance 1728 similar to wearable extended reality appliance 1528, andso forth. In some examples, computational interface device 1700 mayinclude elements of computational interface devices 1500, 1600, 1800,2000, 2100, and 2200 that are not described herein in relation tocomputational interface device 1700 and/or that are not shown in FIG.17.

Integrated computational interface device 1700 may include a foldableprotective cover 1714. Foldable protective cover 1714 may incorporateimage sensor 1710. Foldable protective cover 1714 may be connected tohousing 1702 by a connector 1716. Foldable protective cover 1714 mayinclude multiple sectors, such as a first sector 1718, a second sector1720, and a third sector 1722. Sectors 1718, 1720, and 1722 may enablefoldable protective cover 1714 to stand in the second foldingconfiguration.

Foldable protective cover 1714 may include predefined folding lines1730, which are shown in FIG. 17 as dashed lines. Predefined foldinglines 1730 may separate foldable protective cover 1714 into sectors1718, 1720, and 1722. Some predefined folding lines 1730 may extendacross a full distance of foldable protective cover 1714 and may beparallel to one another. Other predefined folding lines 1730 may extendin different directions. A pattern may be created by predefined foldinglines 1730 to enable foldable protective cover 1714 to be folded intoone or more three-dimensional shapes.

FIG. 18 is a front perspective view of a fourth exemplary embodiment ofan integrated computational interface device 1800 with a foldableprotective cover 1814 in a first folding configuration. FIG. 18 is anexemplary representation of just one embodiment, and it is to beunderstood that some illustrated elements might be omitted and othersadded within the scope of this disclosure. As shown in FIG. 18, foldableprotective cover 1814 may form a three-dimensional shape. For ease ofdescription, elements shown in FIG. 18 that are similar to elementsshown in FIG. 15 are referenced with similar reference numerals. In someexamples, computational interface device 1800 may include elements ofcomputational interface devices 1500, 1600, 1700, 2000, 2100, and 2200that are not described herein in relation to computational interfacedevice 1800 and/or that are not shown in FIG. 18.

Foldable protective cover 1814 may include multiple sectors, such as afirst sector 1818, a second sector 1820, and a third sector 1822.Foldable protective cover 1814 may include one or more flexible sections1832 that enable folding foldable protective cover 1814. Foldableprotective cover 1814 may include a plurality of separated elements,such as one or more triangular elements 1834 and one or more polygonalelements 1836. The plurality of separated elements 1834, 1836 may beconnected to each other by flexible sections 1832.

FIG. 19 is an exploded view of a portion of an exemplary embodiment of afoldable protective cover 1914. FIG. 19 is an exemplary representationof just one embodiment, and it is to be understood that some illustratedelements might be omitted and others added within the scope of thisdisclosure. For ease of description, elements shown in FIG. 19 that aresimilar to elements shown in FIG. 15 are referenced with similarreference numerals.

Foldable protective cover 1914 may include a first external layer 1940,an intermediate layer 1942, and a second external layer 1944. Whenfoldable protective cover 1914 is in a first folding configuration,first external layer 1940 may face the housing of the integratedcomputational interface device and second external layer 1944 may be onan opposite side of foldable protective cover 1914, such that secondexternal layer 1944 faces away from the housing. In some embodiments,intermediate layer 1942 may be located between first external layer 1940and second external layer 1944, to “sandwich” intermediate layer 1942between first external layer 1940 and second external layer 1944 tosupport intermediate layer 1942.

First external layer 1940 and second external layer 1944 may be madefrom a single continuous material, such as a soft material (e.g., anon-woven textile). First external layer 1940 may include an opening1952 such that an image sensor 1910 may receive image data without beingobstructed by first external layer 1940.

Intermediate layer 1942 may be made of a plurality of separated elements1946. In some embodiments, each separated element 1946 may be enclosedin a pocket created by first external layer 1940 and second externallayer 1944, such that the separated elements 1946 do not contact oneanother.

The plurality of separated elements 1946 may be made of a rigid orsemi-rigid material such that the plurality of separated elements 1946may support an image sensor housing 1912 that contains at least oneimage sensor 1910. Image sensor housing 1912 may be configured tointeract with a support element 1948 that is connected to one of theplurality of separated elements 1946 to enable the image sensor housing1912 to be connected to support element 1948. A wire 1950 may beconnected to image sensor 1910 to provide power to image sensor 1910and/or to receive image data from image sensor 1910. While one wire 1950is shown in FIG. 19, more than one wire may be provided, with one wireproviding power to image sensor 1910 and a second wire receiving imagedata from image sensor 1910.

In some embodiments, the integrated computational interface device mayfurther include a wire port in the housing located at a front side ofthe integrated computational interface device opposite the first side ofthe protective cover, the wire port being configured for receiving awire extending from a wearable extended reality appliance. Such a wireport may, for example, enable connection of the wearable extendedreality appliance in a manner so that the wire does not interfere withthe keyboard and/or the input device during use. Having the wire port inthe front side may be preferred to having the wire port in a left sideor right side of the integrated computational interface device, forexample to avoid disturbance of a hand of the user by the wire (ofdisturbance of the wire by the hand) while interacting with the keyboardand/or the input device. Furthermore, the wire port may be located inthe middle or nearly in the middle of the front side (for example, atmost 1 inch from the middle, at most 2 inches from the middle, at most 3inches from the middle, at least one inch from both the left and theright edges of the front side, at least two inches from both the leftand the right edges of the front side, at least 3 inches from both theleft and the right edges of the front side, at least 5 inches from boththe left and the right edges of the front side, closer to the middlethan to any one of the left and the right edges of the front side, andso forth). In some embodiments, as previously described, the wire may beany type of wire suitable for providing power and/or data between theintegrated computational interface device and the wearable extendedreality appliance. For example, the wire may be a Universal Serial Bus(USB) type wire with appropriate connectors for the wire port and thewearable extended reality appliance if the wire is detachable from theintegrated computational interface device and/or the wearable extendedreality appliance. In some embodiments, the wire may extend from anypart of the wearable extended reality appliance. In some embodiments,the wire may be fixedly attached to the wearable extended realityappliance or may be detachable from the wearable extended realityappliance. In some embodiments, the wire may be fixedly attached to theintegrated computational interface device or may be detachable from theintegrated computational interface device.

Referring back to FIG. 15, housing 1502 may include a wire port 1524which may be configured to receive a wire 1526 extending from a wearableextended reality appliance 1528. FIG. 15 shows wearable extended realityappliance 1528 as a pair of smart glasses, but wearable extended realityappliance 1528 may include other forms (e.g., a pair of goggles). Theform of wearable extended reality appliance 1528 does not alteroperation of the embodiment shown in FIG. 15.

The at least one image sensor may in general include any number of imagesensors. In some embodiments, the at least one image sensor may includeat least a first image sensor and a second image sensor. In someembodiments, the first image sensor and the second image sensor may belocated proximate to each other. In some embodiments, the first imagesensor and the second image sensor may be contained in a same imagesensor housing. In some embodiments, the first image sensor and thesecond image sensor may be separated from each other. In someembodiments, the first image sensor and the second image sensor may becontained in separate image sensor housings.

In some embodiments, in the second folding configuration, a field ofview of at least one of the image sensors may be configured to capturean image (at least a partial image) of the user using the device. Forexample, given expected distances between a user and the interfacedevice, the image sensor may be configured so that its field of view istailored primarily to portions of the user relevant to operation ofdisclosed embodiments. For example, in some embodiments, in the secondfolding configuration, a first field of view of the first image sensormay be configured to capture a face of the user while the user types onthe keyboard. Again, the protective cover may be constructed so that theimage sensor may be angled upward toward an expected location of a user.In another example, in the second folding configuration, a first fieldof view of the first image sensor may be configured to capture a face ofthe user while the user physically interacts with the input device.

In some embodiments, the image sensors may be configured such that, whenone of the image sensors is configured to capture an image of the user'sface, another image sensor may be configured to capture another view ofthe user. For example, an image of one or both of the user's hands onthe keyboard. Thus the cover may support one image sensor angled to theexpected location of a user's face while a second image sensor in thesame cover (or at another location on the housing) may be angled tocapture the user's hand movements on the keyboard. For example, in someembodiments, in the second folding configuration, a second field of viewof the second image sensor may be configured to capture an image of thehands of the user while the user types on the keyboard. In anotherexample, in the second folding configuration, the second field of viewof the second image sensor may be configured to capture an image of thehands of the user while the user physically interacts with the inputdevice. In some embodiments, the first field of view and the secondfield of view may at least partially overlap. In some embodiments, thefirst field of view and the second field of view may not overlap.

FIG. 20 is a right side view of a fifth exemplary embodiment of anintegrated computational interface device 2000. FIG. 20 is an exemplaryrepresentation of just one embodiment, and it is to be understood thatsome illustrated elements might be omitted and others added within thescope of this disclosure. For ease of description, elements shown inFIG. 20 that are similar to elements shown in FIG. 15 are referencedwith similar reference numerals. For example, integrated computationalinterface device 2000 may include housing 2002 similar to housing 1502,and so forth. In some examples, computational interface device 2000 mayinclude elements of computational interface devices 1500, 1600, 1700,1800, 2100, and 2200 that are not described herein in relation tocomputational interface device 2000 and/or that are not shown in FIG.20.

Integrated computational interface device 2000 may include a foldableprotective cover 2014. Foldable protective cover 2014 may includemultiple sectors, such as a first sector 2018, a second sector 2020, anda third sector 2022. Sectors 2018, 2020, and 2022 may enable foldableprotective cover 2014 to stand in the second folding configuration, asshown in FIG. 20.

Third sector 2022 of foldable protective cover 2014 may include a firstimage sensor 2060 having a first field of view 2062 and a second imagesensor 2064 having a second field of view 2066. First field of view 2062may enable first image sensor 2060 to capture one or more images of aface of a user of integrated computational interface device 2000. Secondfield of view 2066 may enable second image sensor 2064 to capture one ormore images of hands of the user while the user types on keyboard 2008.

In some embodiments, the at least one image sensor may be connected toat least one gimbal configured to enable the user to change an angle ofthe at least one image sensor without moving the protective cover. Thegimbal may be a pivoted support that enables the at least one imagesensor to rotate about an axis, thereby changing the angle of the atleast one image sensor. In some embodiments, the gimbal may beimplemented as a ball and socket joint to enable the at least one imagesensor to be rotated within a 360° range. In some embodiments, thegimbal may be implemented as a pin joint to enable the at least oneimage sensor to be rotated within one degree of freedom (i.e., eitherside-to-side or up-and-down). In some embodiments, other types ofmechanical joints or connectors may be implemented to enable the atleast one image sensor to be angled within one or more degrees offreedom. In some embodiments, the at least one image sensor may becontained in an image sensor housing that may be connected to the atleast one gimbal. Changing the angle of the at least one image sensormay also change the field of view of the at least one image sensor. Insome embodiments, the gimbal may enable the at least one image sensor tomove side-to-side, up-and-down, or within a 360° range.

FIG. 21 is a right side view of a sixth exemplary embodiment of anintegrated computational interface device 2100. FIG. 21 is an exemplaryrepresentation of just one embodiment, and it is to be understood thatsome illustrated elements might be omitted and others added within thescope of this disclosure. For ease of description, elements shown inFIG. 21 that are similar to elements shown in FIG. 15 are referencedwith similar reference numerals. For example, integrated computationalinterface device 2100 may include housing 2102 similar to housing 1502,and so forth. In some examples, computational interface device 2100 mayinclude elements of computational interface devices 1500, 1600, 1700,1800, 2000, and 2200 that are not described herein in relation tocomputational interface device 2100 and/or that are not shown in FIG.21.

Integrated computational interface device 2100 may include a foldableprotective cover 2114. Foldable protective cover 2114 may includemultiple sectors, such as a first sector 2118, a second sector 2120, anda third sector 2122. Sectors 2118, 2120, and 2122 may enable foldableprotective cover 2114 to stand in the second folding configuration, asshown in FIG. 21.

Third sector 2122 of foldable protective cover 2114 may include an imagesensor 2170 mounted on a gimbal 2172. Gimbal 2172 may be configured toenable the user of integrated computational interface device 2100 tochange an angle of image sensor 2170 without moving foldable protectivecover 2114.

In some embodiments, the integrated computational interface device mayfurther include a holder in the non-key region of the housing (or in thenon-input region of the housing). As previously described, in someembodiments, the holder may be a depression that extends below a planeof at least a portion of the housing or may rest entirely above thesurface of the housing. In some embodiments, the holder may include oneor more structural features to selectively engage with one or more itemssuch as writing instruments, wires or cables, dongles, papers (i.e., topermit the holder to function like a copy stand), goggles, glasses, orother items that the user may wish to easily access or store.

In some embodiments, the holder may be configured for selectiveengagement with and disengagement from a wearable extended realityappliance. As previously described, in some embodiments and shown forexample in FIGS. 6-12B, structural features of the holder may beconfigured to selectively engage with the wearable extended realityappliance such that the wearable extended reality appliance may besnap-fit, force-fit, or pressure-fit into at least a portion of theholder.

In some embodiments, when the wearable extended reality appliance isselectively engaged with the housing via the holder, the wearableextended reality appliance may be connected to and may be transportablewith the keyboard (or with the input device). As previously described,in some embodiments, when the wearable extended reality appliance isselectively engaged with the housing via the holder, the wearableextended reality appliance may be transportable with the housing bybeing securely connected to the housing via the holder.

In some embodiments, the first folding configuration may be associatedwith two encasing modes. In some embodiments, in a first encasing mode,when the wearable extended reality appliance is engaged with the housingvia the holder, the protective cover may overlie the wearable extendedreality appliance and the keyboard. As previously described, in someembodiments in the first encasing mode, the protective cover may providebump, shock, or impact protection to the keyboard and/or the housingand/or the wearable extended reality appliance when the wearableextended reality appliance is selectively engaged with the holder. Insome embodiments, when in the first encasing mode, the protective covermay form a three-dimensional shape extending above an upper surface ofthe housing to form a compartment for selectively enclosing the wearableextended reality appliance. In some embodiments, in a first encasingmode, when the wearable extended reality appliance is engaged with thehousing via the holder, the protective cover may overlie the wearableextended reality appliance and the input device.

In some embodiments, in the first encasing mode of the first foldingconfiguration, a distance of the at least one image sensor from thekeyboard may be between 2 centimeters (cm) and 5 cm. In someembodiments, the three-dimensional shape formed by the protective coverin the first folding configuration may elevate the at least one imagesensor above the keyboard. In some embodiments, the at least one imagesensor may be closer to the keyboard than 2 cm. In some embodiments, theat least one image sensor may be farther from the keyboard than 5 cm. Insome embodiments, in the first encasing mode of the first foldingconfiguration, a distance of the at least one image sensor from theinput device may be between 2 cm and 5 cm.

In some embodiments, in a second encasing mode, when the wearableextended reality appliance is disengaged from the housing, theprotective cover may overlie the keyboard (or the input device). In someembodiments, in the second encasing mode, the protective cover may liesubstantially flat across an upper surface of the housing such that theprotective cover is proximate to the keyboard (or the input device). Insome embodiments, the flexible sections of the protective cover mayenable the protective cover to form both the first encasing mode and thesecond encasing mode. In some embodiments, in the second encasing modeof the first folding configuration the distance of the at least oneimage sensor from the keyboard may be between 1 millimeter (mm) and 1cm. In some embodiments, when the protective cover is in the secondencasing mode, it may lie substantially flat across the upper surface ofthe housing such that the at least one image sensor may be located closeto the keyboard. In some embodiments, the at least one image sensor maybe closer to the keyboard than 1 mm. In some embodiments, the at leastone image sensor may be farther from the keyboard than 1 cm. In someembodiments, in the second encasing mode of the first foldingconfiguration, the distance of the at least one image sensor from theinput device may be between 1 mm and 1 cm.

In some embodiments, in the second folding configuration, the distanceof the at least one image sensor from the keyboard may be between 4 cmand 8 cm. As previously described, when the protective cover is in thesecond folding configuration, the at least one image sensor may bepositioned to face the user of the integrated computational interfacedevice. In some embodiments, the at least one image sensor may be closerto the keyboard than 4 cm. In some embodiments, the at least one imagesensor may be farther from the keyboard than 8 cm. In some embodiments,in the second folding configuration, the distance of the at least oneimage sensor from the input device may be between 4 cm and 8 cm.

In some embodiments, the protective cover may include a notch configuredto hold the wearable extended reality appliance in the first foldingconfiguration when the wearable extended reality appliance isselectively engaged with the housing. In some embodiments, the notch maybe an indentation in the first external layer of the protective coverthat faces the housing when the protective cover is in the first foldingconfiguration. In some embodiments, the at least one image sensor may bepositioned in the notch. In some embodiments, the notch may be locatedon a protrusion from the surface of the protective cover that faces thehousing when the protective cover is in the first folding configuration.For example, the protrusion may be an image sensor housing as previouslydescribed and the notch may include one or more indentations in theimage sensor housing.

FIG. 22 is a front perspective view of a seventh exemplary embodiment ofan integrated computational interface device 2200. FIG. 22 is anexemplary representation of just one embodiment, and it is to beunderstood that some illustrated elements might be omitted and othersadded within the scope of this disclosure. For ease of description,elements shown in FIG. 22 that are similar to elements shown in FIG. 15are referenced with similar reference numerals. For example, integratedcomputational interface device 2200 may include keyboard 2208 similar tokeyboard 1508, wire port 2224 similar to wire port 1524, and so forth.In one example, computational interface device 2200 may be linked withwire 2226 similar to wire 1526, and so forth. In some examples,computational interface device 2200 may include elements ofcomputational interface devices 1500, 1600, 1700, 1800, 2000, and 2100that are not described herein in relation to computational interfacedevice 2200 and/or that are not shown in FIG. 22.

Integrated computational interface device 2200 may include a foldableprotective cover 2214. Foldable protective cover 2214 may includemultiple sectors, such as a first sector 2218, a second sector 2220, anda third sector 2222. Sectors 2218, 2220, and 2222 may enable foldableprotective cover 2214 to stand in the second folding configuration, asshown in FIG. 22.

Third sector 2222 of foldable protective cover 2214 may include a notch2280 configured to accommodate an upper portion of wearable extendedreality appliance 2228 when foldable protective cover 2214 is in thefirst folding configuration such that wearable extended realityappliance 2228 fits at least partially within notch 2280. Notch 2280 maybe three-dimensional in that it may extend into sector 2222. While notch2280 is shown in FIG. 22 as having a rectangular shape, notch 2280 maybe in different shapes to accommodate at least a portion of wearableextended reality appliance 2228 when foldable protective cover 2214 isin the first folding configuration.

Image sensor 2210 may be positioned in notch 2280 such that whenfoldable protective cover 2214 is in the second folding configuration,an optical axis of image sensor 2210 may generally face a user ofintegrated computational interface device 2200. In some embodiments,image sensor 2210 may be flush with a surface of notch 2280 such thatimage sensor 2210 does not extend beyond notch 2280 and when foldableprotective cover 2214 is in the first folding configuration, imagesensor 2210 does not contact wearable extended reality appliance 2228.

In some embodiments, a case for an integrated computational interfacedevice may include at least one image sensor and a foldable protectivecover incorporating the at least one image sensor, wherein theprotective cover may be configured to be manipulated into a plurality offolding configurations. The at least one image sensor and the foldableprotective cover were previously described in greater detail. Theseelements are configured and function in a similar manner as thatpreviously described.

In some embodiments, the case may be connected to the integratedcomputational interface device. As an example, the case may be connectedto the integrated computational interface device by one or more magnets.In some embodiments, one or more magnets may be provided in the case andthe integrated computational interface device may include aferromagnetic material to enable interaction with the magnets such thatthe case may be connected to the integrated computational interfacedevice. As another example, the integrated computational interfacedevice may include a ferromagnetic material and the case may include oneor more magnets to connect the integrated computational interface deviceand the case.

As another example, the case may be attached or coupled to theintegrated computational interface device by a hinge mechanism. Any typeof hinge mechanism known in the art may be used. Although not arequirement, in some embodiments, the hinge mechanism may be made of aflexible material, with a first edge of the flexible material fixedlyattached to the case and a second edge, opposite the first edge, fixedlyattached to the integrated computational interface device. In someembodiments, the hinge mechanism may include one or more knuckles (e.g.,rigid or semi-rigid loop structures) connected to and extending awayfrom the case, one or more knuckles connected to and extending away fromthe integrated computational interface device, and a pin insertedthrough all of the knuckles to create the hinge, such that at least theprotective cover may rotate or pivot around the pin. When assembledtogether prior to inserting the pin, the knuckles of the case and theknuckles of the integrated computational interface device may be alignedor interleaved.

In some embodiments, the integrated computational interface device maybe insertable into at least a portion of the case. For example, the casemay include a lower portion that fits around a lower portion of ahousing of the integrated computational interface device. In someembodiments, the lower portion of the case may be separated from theprotective cover. In some embodiments, the lower portion of the case maybe connected to the protective cover. In some embodiments, the lowerportion of the case may be made of a rigid material, such as plastic. Insome embodiments, the lower portion of the case may be made of aflexible material, such as fabric, neoprene, or other elastomericmaterial that may stretch over the lower portion of the housing.

In some embodiments, in a first folding configuration, the protectivecover is configured to encase a housing of the integrated computationalinterface device having a key region and a non-key region. The housing,the key region, the non-key region, and the first folding configurationwere previously described in connection with other embodiments. Theseelements are configured and function in a similar manner as thatpreviously described. In some embodiments, in a first foldingconfiguration, the protective cover is configured to encase a housing ofthe integrated computational interface device having an input region anda non-input region.

In some embodiments, in a second folding configuration, the protectivecover is configured to stand in a manner that causes an optical axis ofthe at least one image sensor to generally face a user of the integratedcomputational interface device while the user types on a keyboardassociated with the key-region. The configuration of the protectivecover in the second folding configuration and the optical axis of the atleast one image sensor were previously described in connection withother embodiments. These elements are configured and function in asimilar manner as that previously described. In some embodiments, in asecond folding configuration, the protective cover is configured tostand in a manner that causes an optical axis of the at least one imagesensor to generally face a user of the integrated computationalinterface device while the user physically interacts with an inputdevice associated with the input region.

Some disclosed embodiments may involve changing display of virtualcontent based on temperature. Virtual content may include anyinformation as defined elsewhere in this disclosure. Changing display ofvirtual content may refer to modifying or altering in some way one ormore characteristics associated with displayed virtual content. Suchcharacteristics may include color scheme, opacity, intensity,brightness, frame rate, display size, virtual object type, or any otherparameters that may affect how a user views the virtual contentpresented to the user.

In some embodiments, the virtual content may include text inputted by akeyboard connectable to the wearable extended reality appliance. Akeyboard may include a panel of keys that may allow a user to input oneor more alphanumeric characters that may include words, phrases,sentences, paragraphs, or other textual matter. In addition to inputtingtext, the keyboard may include one or more keys that may allow the userto input numbers, symbols, or keyboard strokes configured by the user.This content may also be displayed as a virtual object that is part ofdisplayed virtual content.

Virtual content may include one or more virtual objects. For example,virtual content may be a document or a video. As part of the presentedvirtual content, a virtual object associated with the virtual object maybe a search bar, task bar, brightness adjustment bar or any otherelement. In one example illustrated in FIG. 1, virtual content mayinclude virtual display 112 and/or virtual widgets 114A, 114B, 114C,114D, and 114E.

The keyboard may be connected to the wearable extended reality appliancevia a wire 108 (see FIG. 1) or wirelessly through Bluetooth or otherwireless protocol. When the wearable extended reality appliance userenters text or another command via the keyboard, the text may show upvia a virtual screen 112 (see FIG. 1) that is displayed close to theuser. The text may also show up via a physical screen or other physicalobject depending on the wearable extended reality appliance user'spreference and the user's environment. By way of example, FIG. 1illustrates a keyboard 104 that may be connected to a wearable extendedreality appliance 110.

Some disclosed embodiments may include displaying virtual content via awearable extended reality appliance, wherein during the displaying ofthe virtual content, heat is generated by at least one component of thewearable extended reality appliance. A virtual reality appliance, asdescribed elsewhere in this disclosure includes electronics and opticsfor displaying augmented content to a wearer of the appliance. Forexample, such an appliance may include a frame, at least one lens, oneor more heat-emitting light sources configured to project images on theat least one lens, one or more processing devices, one or more wirelesscommunication devices, and/or other devices (e.g., resistors, inductors,capacitors, diodes, semiconductor devices, or other circuitry) that maybe necessary for the operation of the wearable extended realityappliance. At least some of the components of the wearable extendedreality appliance may generate heat. For example, components like lightemitting sources, processing devices, wireless communication devices,resistors, inductors, capacitors, diodes, and/or other circuitry maygenerate heat during operation of the wearable extended realityappliance. Because of such heat generation, when the wearable extendedreality appliance is used continuously, there may be a risk ofoverheating of one or more components of the wearable extended realityappliance, or a risk of reaching a maximal allowed working temperatureof the wearable extended reality appliance, for example, due toregulatory requirements, to avoid risk of harm to the user, and/or topreserve the life of the components. One possible solution may includeturning off one or more components of the wearable extended realityappliance to reduce the amount of heat generated, thereby reducing therisk of overheating or reaching the maximal allowed working temperatureof the wearable extended reality appliance. However, if the lightsources in the wearable extended reality appliance were to be turned-offto permit cool-down, the wearable extended reality appliance would losefunctionality. In one example, a processing device included in thewearable extended reality appliance may be configured to render virtualcontent for presentation, and if this processing device were to beturned-off to permit cool-down, the wearable extended reality appliancewould lose functionality. In another example, a communication deviceincluded in the wearable extended reality appliance may be configured toreceive virtual content for presentation, and if the communicationdevice were to be turned-off to permit cool-down, the wearable extendedreality appliance would lose functionality. In yet another example, amemory device included in the wearable extended reality appliance may beconfigured to store and provide virtual content for presentation, and ifthe memory device were to be turned-off to permit cool-down, thewearable extended reality appliance would lose functionality.Accordingly, other methods of reducing the risk of overheating orreaching the maximal allowed working temperature of the wearableextended reality appliance without loss of functionality are disclosedherein.

In some embodiments, the heat may be generated (at least partly) by aplurality of heat-emitting light sources included in the wearableextended reality appliance, and the operations performed by the at leastone processor further include modulating a set of working parameters ofat least one of the heat-emitting light sources. As discussed above, awearable extended reality appliance consistent with embodiments of thepresent disclosure may include one or more light sources. Such lightsources may be powered via one or more power sources (e.g., a battery)that may supply electrical energy in the form of a voltage and/orcurrent. Part of the electrical energy may be converted into light bythe light sources. Some or all of the electrical energy supplied to thelight sources may be dissipated as heat. Thus, the amount of heatdissipated by the light source may depend on one or more parameters suchas current, voltage, power, and/or duration of time for which electricalpower is supplied to the light source. Accordingly, the amount of heatdissipated by the light source may be reduced by modulating one or moreof these parameters (e.g., voltage, current, duration of time)associated with the electrical energy supplied to the light source.Modulating the one or more parameters may involve adjusting some or allparameters. In some embodiments, modulating the one or more parametersmay involve reducing values of all the parameters. In some embodiments,modulating the one or more parameters may involve reducing values ofsome parameters while maintaining the values of the other parameters orincreasing the values of the other parameters. For example, the amountof heat dissipated by the light source may be reduced by reducing atotal amount of electrical power supplied to the light source. Asanother example, the amount of heat dissipated by the light source maybe reduced by maintaining the amount of electrical power being suppliedto the light source while reducing a duration for which the electricalpower is supplied to the light source. By way of another example, theamount of heat dissipated by the light source may be reduced by reducingsome or all of voltage, current, or electrical power being supplied toone or more of the heat-emitting light sources. In another example, theamount of heat dissipated by the light source may be reduced by pulsingthe voltage, current, or electrical power being supplied to one or moreof the heat-emitting light sources. In this example, the voltage,current, or power supplied to the heat-emitting light sources may beturned on or off for a period of time as configured by the wearableextended reality appliance user. In some examples, the set of displaysettings of at least one of the heat-emitting light sources may includeat least one of presentation resolution, frame rate, brightness,opacity, display size, or a color scheme. In one example, thepresentation resolution may be reduced to reduce the heat generated bythe plurality of heat-emitting light sources. In another example, thepresentation frame rate may be reduced to reduce the heat generated bythe plurality of heat-emitting light sources. In yet another example,the brightness may be reduced to reduce the heat generated by theplurality of heat-emitting light sources. In an additional example, theopacity may be reduced to reduce the heat generated by the plurality ofheat-emitting light sources. In yet another example, the display sizemay be reduced to reduce the heat generated by the plurality ofheat-emitting light sources. In an additional example, a color scheme ofa presented virtual content may be changed to a favorable color schemeto reduce the heat generated by the plurality of heat-emitting lightsources.

In some embodiments, heat may be generated by at least one processingdevice included in the wearable extended reality appliance. One or moreprocessors may be included in the wearable extended reality appliance,for example to render virtual content for presentation (for example,from a model, from a three-dimensional model, etc.), to control datacommunication, to analyze input captured using one or more sensorsincluded in the wearable extended reality appliance (such as imagesand/or videos captured using one or more image sensors included in thewearable extended reality appliance, motion data captured using one ormore motion sensors, positioning data captured using one or morepositioning sensors), to interpret various inputs (e.g., gestures,textual inputs, and/or audio inputs) to display or share virtualcontent, or for any other purpose in furtherance of the functionality ofthe appliance. A processor may include numerous (e.g., millions orbillions of) microscopic transistors, which may be used to amplify orswitch electrical signals and power. When performing certain commands,such as displaying virtual content, these transistors may either enableelectrical signals to pass through them or may block the signals. When atransistor resists the flow of electricity, heat is generated within theprocessing device. The more complicated the command, the moreelectricity flows through the processor, more transistors may be used toperform the command, and as a result more heat may be generated. Forexample, less heat may be generated by the processing device when adocument with only text is to be displayed, and more heat may begenerated when a video is to be displayed or when a three-dimensionalmodel is rendered for presentation.

Some disclosed embodiments may include modulating a set of workingparameters of the at least one processing device, the set of workingparameters of the at least one processing device including at least oneof voltage, current, power, clock speed, or number of active coresassociated with the at least one processing device. Accordingly, theamount of heat generated by the processing device may be reduced byreducing one or more of the voltage or current associated with theelectrical energy supplied to the processing device, by reducing a totalamount of electrical power supplied to the processing device, byreducing a duration for which the voltage, current, or electrical poweris supplied to the processing device, by reducing a clock speed, or bylimiting a number of active cores in the processing device. In oneexample, the rendering resolution and/or frame rate may be reduced toreduce the burden on the processing device and thereby reduce the heatgenerated by the processing device. In another example, the frequency ofanalysis of data captured using one or more sensors included in thewearable extended reality appliance may be reduced, to thereby reducethe heat generated by the processing device.

Clock speed of a processing device may refer to a number of pulses persecond generated by an oscillator that sets the timing at which varioustransistors in the processing device may be switched on or off. Clockspeed may be generally used as an indicator of a processor's speed.Reducing the clock speed may help reduce the rate at which one or moretransistors in the processing device are switched on/off, which in turnmay reduce the amount of heat generated by the processing device.

Active cores in a processing device may refer to the number ofprocessing units within a processor. Each core may include numeroustransistors which, as described above, generate heat based on howcomplicated a particular task is. The more cores a processor has, themore tasks the processor is capable of completing simultaneously.However, one or more of these cores may be temporarily deactivated.Temporarily deactivating a core results in little to no heat beinggenerated by that core, which in turn may help reduce the amount of heatbeing generated as a whole by the processing device.

In some embodiments, heat is generated by at least one wirelesscommunication device included in the wearable extended realityappliance. As discussed elsewhere in this disclosure the wearableextended reality appliance may include a wireless communication device.In order to communicate with other devices in a network, the wirelesscommunication device may transform incoming electrical energy intoradiofrequency (RF) energy. The wireless communication device may emitthe RF energy. The wireless communication device may include atransmitter, an encoder, and a receiver, which include electroniccomponents such as transistors and resistors. Operation of some or allof these wireless communication device components may dissipate some ofthe electrical energy and/or the RF energy as heat.

In other embodiments, when heat is generated by at least one wirelesscommunication device included in the wearable extended realityappliance, processor operations may further include modulating a set ofworking parameters of the at least one wireless communication device,the set of working parameters of the wireless communication deviceincluding at least one of signal strength, bandwidth or amount oftransmitted data.

Signal strength may refer to the transmitter power output as received bya reference antenna at a distance from the transmitting antenna. Here,the transmitting antenna may be a first wearable extended realityappliance sharing content, and the receiving antenna would be a secondwearable extended reality appliance receiving the shared content.Bandwidth may refer to the maximum capacity of a wireless communicationslink to transmit data over a network connection in a given amount oftime. In the context of some embodiments, bandwidth may refer to theamount of data that may be transmitted by multiple wearable extendedreality appliances in a single network. The amount of transmitted datamay refer to the amount of data that a single wearable extended realityappliance may present or share in the form of virtual content.Modulating these parameters may involve reducing the signal strength,bandwidth, or amount of transmitted data so as to reduce the amount ofpower used to transform electrical energy into RF energy. Because theamount of power used to transform the electrical energy is reduced, theamount of heat generated by the wireless communication device may alsobe also reduced.

By way of example, FIG. 23 is a block diagram illustrating a wearableextended reality appliance 2310 that may include one or moreheat-emitting light sources 2312, one or more processing devices 2314,one or more wireless communication devices 2316, and/or other electricaland/or mechanical devices as described above. As described herein,heat-emitting light source 2312 may have working parameters includingbut not limited to voltage, current, and/or power. The processing device2314 may have working parameters including but not limited to voltage,current, power, clock speed, and/or number of active cores. The wirelesscommunication device 2316 may have working parameters including but notlimited to signal strength, bandwidth, and/or the amount of transmitteddata.

Some disclosed embodiments may include receiving information indicativeof a temperature associated with the wearable extended realityappliance. Information indicative of a temperature may refer to anumerical value that represents a temperature, or any other temperatureindicator. In some embodiments, the numerical value may be a temperaturevalue, for example, as measured in ° C., ° F., or in any othertemperature units. Alternatively, the information may be a measurementof a physical characteristic that indicates a change in temperature. Forexample, the information may include a value of a parameter such asvoltage, current, resistance, power, or any other physicalcharacteristic that changes with temperature and is thereforerepresentative of a temperature. In some embodiments, the wearableextended reality appliance frame may include a temperature sensorcapable of determining a value of a temperature associated with thewearable extended reality appliance. The temperature sensor associatedwith the wearable extended reality appliance may be one of a variety oftemperature sensors. For example, the temperature sensor may be athermocouple, a resistance temperature detector (RTD), a thermistor, ora semiconductor based integrated circuit.

Some disclosed embodiments may include determining a need to changedisplay settings of the virtual content based on the receivedinformation. As discussed above, one or more of the wearable extendedreality appliance components such as light source, processing device,and wireless communication device, may generate heat during use. Whenthe wearable extended reality appliance becomes too hot, it may notoperate as efficiently. Moreover, excessive heating of the wearableextended reality appliance may reduce the life of the appliance.Additionally, overheating may cause one or more components of thewearable extended reality appliance to fail and/or reaching the maximalallowed working temperature of the wearable extended reality appliancemay injure the wearable extended reality appliance user. Otherembodiments may include determining a need to change display settings ofthe virtual content not based on the received information indicative ofthe temperature associated with the wearable extended reality appliance,for example, based on user input, based on environmental situation,based on time, and/or any other temperature information not associatedwith the wearable extended reality appliance.

Each component has working parameters, as described above, that may bemodulated to reduce an amount of heat being generated by the wearableextended reality appliance. Additionally or alternatively, modifying oneor more virtual content display settings may also reduce the temperatureassociated with the wearable extended reality appliance. Modifying theone or more virtual content display settings may cause one or moreworking parameters of the light source, the processor, and/or wirelesscommunication device to be modulated, thereby reducing an amount of heatgenerated by the wearable extended reality appliance, which in turn mayreduce the associated temperature. For example, reducing a brightness ofdisplayed virtual content may involve reducing one or more of a current,voltage, and/or a power that is delivered to the heat-emitting lightsource, thus reducing the temperature of that component. Other settingsthat may be modified to reduce the temperature may include, for example,brightness, opacity, display size, resolution, rendering details, framerate, or a color scheme being used for the display. Modifying one ormore of these parameters may involve reducing one or more of a current,a voltage, and/or a power that is delivered to the heat-emitting lightsource, thus reducing the temperature of that component. Suchmodifications may be based on either the content being displayed (e.g.,a video or document) and/or the state of the component (i.e., theheat-emitting light source is overheating).

Some disclosed embodiments may include changing display settings of thevirtual content when the temperature associated with the wearableextended reality appliance reaches a threshold associated with theextended reality. In some disclosed embodiments may include changingdisplay settings of the virtual content before the temperatureassociated with the wearable extended reality appliance reaches athreshold associated with the extended reality appliance. Displaysettings that may be changed include those associated with heatgeneration, such as brightness, and displayed frames per second, or anyof the settings discussed above. The need to change display settings ofthe virtual content may be determined based on a threshold temperature.The threshold temperature may refer to a maximum temperature above whicha particular component may be expected to operate inefficiently or fail.Additionally or alternatively, the threshold temperature may refer to atemperature at or above which a reliability or working life of acomponent may be reduced by a predetermined amount. For example, thethreshold temperature may be associated with a reduction in working lifeof a component by 25%, 50%, 75% or any other amount. Additionally oralternatively, the threshold temperature may be set by a manufacturer ofthe wearable extended reality appliance, by a regulatory entity, and/orany organization that would set a specific threshold temperature for aspecific purpose. For example, some organizations may have differentpresentation needs compared to other organizations, and may set amaximum threshold temperature accordingly. Additionally oralternatively, the threshold temperature may be associated with a safetyissue, such as a safety standard or a safety range of temperatures forhead mounted devices. In one example, a processor associated with somedisclosed embodiments may determine that one or more display settingsmust be changed when a temperature associated with the wearable extendedreality appliance is at or above the threshold temperature. In otherexamples, a current temperature associated with the wearable extendedreality appliance may indicate a likelihood that the wearable extendedreality appliance will reach the threshold associated with the wearableextended reality appliance, for example, if a current working loadcontinues for a selected time period and/or if a typical working loadwill take place for a selected time period, and in response the displaysettings of the virtual content may be changed before the temperatureassociated with the wearable extended reality appliance reaches thethreshold associated with the wearable extended reality appliance. Inone example, an ongoing trajectory of the temperature associated withthe wearable extended reality appliance over time may be analyzed todetermine the likelihood that the wearable extended reality appliancewill reach the threshold associated with the extended reality appliance,(e.g., if a current working load continues for a selected time period,and/or if a typical working load will take place for a selected timeperiod) for example using an extrapolation algorithm, regression model,and/or any other data analysis algorithm.

In some embodiments, the threshold associated with the wearable extendedreality appliance may be between 33 and 40 degrees Celsius. In otherexamples, the threshold associated with the wearable extended realityappliance may be below 33 degrees Celsius or above 40 degrees Celsius.In some examples, a value of the temperature threshold may be configuredfor different types of users. For example, there may be a differenttemperature threshold for a child as opposed to an appliance configuredfor an adult.

In some embodiments, the threshold temperature may be configured basedon preferences of a user of the wearable extended reality appliance. Thetemperature threshold for different users may be configured based on theskin sensitivity of a particular user. For example, a child or moreelderly user may have more sensitive skin than a typical adult. As aresult, a younger or elderly user may configure the temperaturethreshold to be relatively lower that the threshold temperature selectedby a typical adult user. As another example, a user may prefer a highertemperature threshold when the ambient temperature is low as compared towhen the ambient temperature is high. In another example, a user mayprefer a lower temperature threshold in summer and a higher temperaturethreshold in winter.

Some disclosed embodiments may involve determining a value for thethreshold based on a user profile associated with a user of the wearableextended reality appliance. A user profile may include one or morecharacteristics of a particular user. The user profile may be configuredin a variety of ways, and the at least one processor may determine athreshold temperature accordingly. For example, a user may input datasuch as age, gender, preferred use of the appliance, how often he or sheuses the appliance, and/or whether the user prefers performance (i.e.,operating at higher temperatures) or comfort (i.e., typically operatingat cooler temperatures). Based on the inputted data, and/or datareceived from other wearable extended reality appliance users, theprocessor may determine the appropriate temperature threshold. In oneexample, based on a user profile of a younger user who prefers peakperformance and uses the appliance often, the threshold temperature maybe closer to 40 degrees Celsius. In another example, based on a userprofile of an older user who prefers comfort and only uses the applianceoccasionally, the threshold temperature may be closer to 33 degreesCelsius. The user profile may be updated by the user at any time. Insome other examples, a value for the threshold may not be based on auser profile associated with a user of the wearable extended reality.

Some disclosed embodiments may include determining a display settingschange for the virtual content regardless of a user of the wearableextended reality appliance and/or of a user profile associated with theuser of the wearable extended reality appliance. For example, the changemay be based on the current displayed content, on manufacturerpresetting, on configuration files associated with the wearable extendedreality appliance (rather than the user), regulatory requirements,and/or any other outside factors.

Some disclosed embodiments may include determining a display settingschange for the virtual content based on a user profile associated with auser of the wearable extended reality appliance. While the aboveembodiment involves determining a threshold temperature based on a userprofile, this embodiment involves changing the display settingsthemselves based on the user profile. In this embodiment, differentusers may have different temperature threshold preferences and/ordifferent virtual content display preferences. The display settingpreferences may refer to a user's default settings regarding colorscheme, brightness, intensity, display size, or any other displaysetting. In this embodiment, the user profile may state that a userprefers to modify brightness in lieu of color scheme. Display settingpreferences are related to the threshold temperature as included in auser's profile. For example, a user that may prefer a lower thresholdtemperature may decrease a brightness of the displayed virtual contentas compared to a user who may prefer a higher threshold temperature, whowould not change those display settings immediately.

By way of example, a wearable extended reality appliance may beconfigured, based on a user profile, to have a threshold temperature of35 degrees Celsius. Thus, one or more display settings associated withthe virtual content may be modified before a temperature of one or morecomponents of the wearable extended reality appliance reaches thethreshold temperature (e.g., 35-degree Celsius). Additionally, the atleast one processor, based on the user profile, may modify one or moredisplay settings, such as brightness or intensity, when a temperature ofone or more components, such as the heat-emitting light source, at leastone processor, or wireless communication device of the wearable extendedreality appliance reaches 34 degrees Celsius.

Temperature information associated with the wearable extended realityappliance may be updated over time. The longer the wearable extendedreality appliance is used, the more temperature information will bestored. The processor may receive this information either continuouslyor periodically. For example, there may be times of the day where theappliance is used more frequently than other times of the day. Thenon-transitory computer readable medium may store this data, which maybe comprised of working parameter data (e.g., voltage, current, power,clock speed, and others), temperature data, time data, display settings,and the type of virtual content, i.e., a document or a video, that isdisplayed at certain times of the day. The processor, based on thecompiled data, may predict how the wearable extended reality appliancemay be used based on the time of day, and adjust the display settingsaccordingly. In some embodiments, the at least one processor maydetermine a need to change display settings before the thresholdtemperature is reached.

Other embodiments may include predicting peak use times based on amachine learning model or other similar algorithm. In this embodiment,test data, such as stored temperature data, feeds into the algorithm sothat the model may predict when the wearable appliance will be used.

In another embodiment, the wearable extended reality appliance maypredict what type of content is to be displayed at which parts of theday based on stored temperature data. For example, displaying videocontent may require more of a specific combination of workingparameters, such as power, compared to displaying a document. In oneexample, displaying a video would thus cause a heat-emitting lightsource to generate more heat than displaying a document, thereforereaching the threshold temperature more quickly. Based on the storedtemperature data as part of the non-transitory computer readable medium,the wearable extended reality appliance may predict the changes in theheat emitted from an appliance component, and preemptively adjustdisplay settings. For example, based on the user's history, the systemmay predict that a video is displayed for at least 30 minutes at 8 PMand may adjust display settings accordingly.

In some embodiments, a degree of change to the display settings of thevirtual content may be based on the temperature indicated by thereceived information. In other examples, the degree of change to thedisplay settings of the virtual content may not be based on thetemperature indicated by the received information (for example, mayalways be the same degree of change, may be based on a user, may bebased on contextual information, may be based on the virtual contentdisplayed, and/or any other change that is not based on receivedtemperature information, i.e., may be based on environmental factors).Degree of change to a display setting may refer to an amount by which aparticular display setting is modified. As described above, temperatureinformation may be received by a processor associated with the wearableextended reality appliance. In some embodiments, when the receivedinformation indicates that the temperature is slightly above (e.g., afew degrees such as 1 to 5 degrees Celsius) the threshold temperature,one or more display settings may be modified accordingly. A slightmodification may involve a 1% to 5% change in a display setting. Forexample, if the temperature threshold of a component of a user'swearable extended reality appliance is 35 degrees Celsius and thetemperature sensor receives information that the component is 36 degreesCelsius, the degree of change in the display settings may in the rangeof 1% to 5%, depending on how the user configures the appliance. Forexample, the brightness of displayed virtual content may be reduced from1% to 5%.

In some embodiments, when the temperature sensor detects that thewearable extended reality appliance temperature is significantly abovethe threshold temperature (e.g., greater than 5 degrees Celsius), thedisplay settings may be more aggressively modified and/or multiplevirtual elements may be modified. A more aggressive degree of change mayinvolve modification up to a 11% to 25% or more change in displaysettings and may affect multiple virtual objects as part of displayedvirtual content. For example, if the temperature threshold of acomponent of a user's wearable extended reality appliance is 35 degreesCelsius and the temperature sensor receives information that thecomponent is 40 degrees Celsius, the display settings may be modified by11% to 25% or more. In this example, a more aggressive modification mayinvolve one or more of reducing the brightness of the displayed contentfrom 11% to 25%, changing the color scheme of a portion of the displayedvirtual content, and/or reducing the frame rate of displayed videocontent.

While some embodiments involve reducing a level of a particular displaysetting, such as a brightness level, different display settingsthemselves may produce more heat than others, and thus may be result ina greater heat reduction when modified. For example, adjusting theintensity of virtual content may have more of a heat reduction effectthan changing the color scheme of the virtual content. Thus, in someembodiments, a second extended reality display parameter may be selectedbased on the first signal regarding a wearable extended realityappliance component's temperature. A display parameter may be acomponent of display settings. For example, displayed virtual contentmay involve multiple display settings, such as brightness, intensity,and/or frame rate. A display parameter may be brightness, or intensity,or frame rate, which together comprise display settings. In thisembodiment, the at least one processor may select a more aggressive atleast one extended reality display parameter for the first signalindicative of a hotter temperature, and a less aggressive at least oneextended reality display parameter for a first signal indicative of acooler temperature. In this context, the more aggressive the displayparameter, the greater impact it may have on the user experience.

In some embodiments, changing the display settings of the virtualcontent may be based on data indicative of temperature trajectory. Thetemperature trajectory may be determined based on the stored temperatureof the extended reality appliance. The data used to determine thetemperature trajectory may be measured in a variety of different ways.In some embodiments, the temperature data may be continuously orperiodically supplied from the temperature sensor and may be stored in astorage location associated with the wearable extended realityappliance. For example, the temperature data may be provided by thetemperature sensor every second, every minute, every 5 minutes, every 10minutes, or at any other desired frequency. In another example,temperature data may be provided by the temperature sensor based onincremental temperature increases, as in a stepwise function. Theseincremental temperature increases may be configured by the user. In thisexample, every time a wearable extended reality appliance componentreaches a certain temperature, such as increasing from 33 degreesCelsius to 35 degrees Celsius, data may be stored in a storage locationor storage device associated with the wearable extended realityappliance.

As described above, the temperature trajectory may be based ontemperature information stored in the non-transitory computer readablemedium over time. The temperature trajectory may be used to determine atime period during which the wearable extended reality appliance mayexperience peak usage. For example, a wearable extended realityappliance user may display video content at certain times of the day,thus generating more heat during those times of the day compared toother times of the day when video content is not displayed. Based onthis usage data, a processor may preemptively adjust display settings atthe times of day associated with peak usage when a temperature of thewearable extended reality appliance is expected to increase.

Some disclosed embodiments may include predicting a time when thewearable appliance may be inactive. This may occur by collecting data onhow the appliance is historically used, and projecting how it will beused in a current for future situation. In addition to historicalbehavioral data, stored temperature data or a machine learningalgorithm, described above, may be used to predict times where theappliance may become inactive. The processor as part of the wearableextended reality appliance may interpret such data to determine when theappliance is likely to become inactive.

In one example, the processor may analyze temperature variation over aperiod of time, such as a work day, to determine what times thetemperature tends to be below a certain value. As stated above, theprocessor may employ a machine learning model or similar algorithm toperform this task. This value may typically be at or within one or twodegrees Celsius of ambient temperature, because the wearable extendedreality will be inactive and thus not generating heat. Additionally, theprocessor may determine periods of inactivity based on the power levelover time. For example, periods of the day where little or no power isused may indicate that the wearable extended reality appliance isinactive. By contrast, periods of the day where the used power levelsare high may indicate periods of peak use. In another example, theprocessor may determine periods of inactivity based on the battery lifeof the wearable extended reality appliance. For example, periods wherethe battery is depleted slowly may indicate periods of non-use, whereasperiods where the battery is depleted quickly may indicate periods ofpeak use.

In another example, the wearable appliance may be inactive at the end ofthe working day, during a face-to-face meeting, or in any other scenariowhere the user is away from his or her workstation. The wearableextended reality appliance may predict these time periods of inactivitybased on one or more of stored temperature information, stored powerusage information, stored battery demand information, or storedinformation associated with any other working parameter of any of thecomponents associated with the wearable extended reality appliance.Additionally, as discussed above, prior behavioral use data may be usedas part of the prediction process.

Some disclosed embodiments may include changing the display settings ofthe virtual content when the heat generated by the wearable extendedreality appliance exceeds a critical value and the predicted timeexceeds a critical duration. A critical value may be a threshold amountof heat above which the temperature may exceed the threshold temperatureconfigured by the user or the threshold temperature that determinesreliability of or usable life of one more components. In one example, ifthe appliance is generating too much heat, such as when video content isbeing displayed at full brightness and intensity settings, the criticalvalue may be reached more quickly. A critical duration may refer to theamount of time that virtual content may be displayed without anymodification. In the above embodiments, critical duration may relate tothe predicted duration or time of activity. Thus, when the amount ofheat generated exceeds the critical value or the device is active duringa predicted period of inactivity, the display settings need to bechanged to prevent overheating or exceeding the threshold temperature.Indeed, when the wearable extended reality appliance is generating moreheat, the critical duration may be short, and when the wearable extendedreality appliance is generating less heat, the critical duration may belonger.

In the above embodiments, the at least one processor may predict thetime that the wearable extended reality appliance will be inactive. Ifthis predicted time is less than the critical duration as configured bythe appliance user, i.e., the appliance is active for longer than thecritical duration, the at least one processor may change the displaysettings in order to reduce the amount of heat generated by the wearableextended reality appliance component. If the processor does not changethe display settings, the temperature of one or more components mayexceed the associated threshold temperature. If the predicted time doesnot exceed the critical duration, however, no change in display settingsis required.

Some disclosed embodiments may include maintaining current displaysettings when the heat generated by the wearable extended realityappliance exceeds a critical value and the predicted time is below acritical duration.

In some embodiments, display settings may not be adjusted even thoughexcess heat is generated, given that a period of expected inactivity isimminent. For example, a wearable extended reality appliance user may begiving a presentation towards the end of a typical work day (based onthe usage history). Even though the presentation would generate heat,the wearable extended reality appliance might not modify the displaysettings, expecting that the appliance will soon be inactive.

The at least one processor may also modify display settings based on howmuch battery is left in the wearable extended reality appliance. Forexample, presenting certain virtual content, such as a video, mayrequire a certain amount of battery life in order to be presentedwithout modifying any display settings. The wearable extended realityappliance may compare how much battery life is required for displayingcertain content to how much battery life is left in the wearableextended reality appliance and may modify the display settingsaccordingly. In one example, if presenting the virtual content requiresmore battery life than is currently left in the appliance, the displaysettings may be modified.

Some disclosed embodiments may include, based on the determination thatdisplay settings need to be changed, changing the display settings ofthe virtual content to achieve a target temperature. For example, if theinformation received tends to establish that a threshold heat level willbe exceeded, one or more processors might determine that display settingchanges are needed.

The determined need may relate to a single display parameters ormultiple display parameters, more than one of which may be modified atonce. While displaying the content using the first at least one extendedreality display parameter, a first signal related to one or morewearable extended reality appliance components, such as theheat-emitting light sources, processing device, or wirelesscommunication device, may be received. This first signal may be used todetermine a need to reduce the temperature of at least one wearableextended reality appliance component. In response to the need to reducethe temperature of the at least part of the wearable extended realityappliance, the content may be displayed via a wearable extended realitydevice using a second at least one extended reality display parameter.For example, brightness may be adjusted based on received temperaturedata.

In one example, two display parameters may be adjusted to reduce heatgeneration. The second extended reality display parameter, for examplecolor scheme, may differ from the first extended reality displayparameter, for example brightness, and the second at least one extendedreality display parameter may be configured to reduce heat output of thewearable extended reality device.

In the context of some embodiments, a virtual element may refer to oneor more portions or an entirety of one or more items displayed in theextended reality environment. For example, a virtual element may be avirtual object or part of a virtual object being displayed in theextended reality environment. By way of an example, when virtual contentincludes multiple documents, a virtual object may be one of the multipledocuments. A virtual element may include the virtual object itself,e.g., the document, or it may include a portion of the virtual object,such as, the task bar, navigation pane, styles tab, or any other part ofthe document.

Modifying one or more of the display settings described above may reducea temperature of the wearable extended reality appliance, and thereforebring the appliance closer to a target temperature. The targettemperature may refer to a desired or preferred operating temperature ofthe wearable extended reality appliance. The preferred operatingtemperature may be configured as such to ensure peak performance, toprevent malfunctioning of the appliance, to improve reliability, toprolong the usable life of the appliance, and/or to be the mostcomfortable for the user.

In some embodiments, the target temperature may refer to the thresholdtemperature at which display settings are modified. The targettemperature may be configured based on a user's preference and maydiffer based on the type of user. For example, the target temperaturemay be lower for a child, i.e., a more sensitive user, than for anadult.

For example, the wearable extended reality appliance may be displayingvirtual content when a processor associated with the wearable extendedreality appliance determines that a temperature of one or morecomponents of the wearable extended reality appliance has reached athreshold temperature. In response, the processor my decrease abrightness of the content (i.e., a first extended reality displayparameter). As stated above, display settings may comprise multipleextended reality display parameters. In some embodiments, one or moreadditional extended reality display parameters may be modified dependingon the temperature of associated with the wearable extended realityappliance. For example, a color scheme or a frame rate of presentedvideo may be modified in addition to the brightness of the presentedcontent when the temperature of one or more components of the wearableextended reality appliance reaches or exceeds the threshold temperature.

By way of example, FIG. 24 illustrates how the temperature 2410 of thewearable extended reality appliance changes over time 2412. Theappliance may be configured to change display settings 2414 to preventthe temperature from reaching a threshold temperature 2416. Further,after the temperature of the wearable extended reality appliance returnsto acceptable limits, the appliance may be configured to alter displaysettings to the initial value 2418.

In some examples, the extended reality display parameter to be modifiedmay be selected based on the virtual content. For example, certainvisual display parameters may be modified when the virtual contentincludes video content, and other extended reality display parametersmay be modified when the virtual content includes textual content. Inthis example, when video content is presented, the frame rate may bereduced, and when documents are presented, the brightness may bereduced.

Certain display parameters may require more active working parameters,i.e., may have higher power or voltage demands to display content, ascompared to other display parameters. Such a more demanding extendedreality display parameter may be brightness, intensity, display size, orframe rate. Other display parameters may not require as many workingparameters and may include color.

Changing the display settings may occur in a many different ways. Insome embodiments, changing the display settings of the virtual contentmay include modifying a color scheme of at least a portion of thevirtual content. A color scheme may refer to a color palette of thevirtual content, a color temperature, shades of one color or a subset ofcolors. The color scheme may be modified by changing one or more of thecolor palette, the color temperature, a number of available shades of aparticular color, or changing the displayed colors to grayscale or blackand white. Displaying virtual content in an original color scheme mayuse more processing power, and thus generate more heat, than displayingvirtual content with a modified color scheme.

Virtual content may have a variety of different display parameters. Insome embodiments, the at least one processor may be configured to modifya color scheme of some or all portions of virtual content. For example,the at least one processor may modify the color scheme associated with afirst portion of virtual content displayed in an extended realityenvironment and/or via the wearable extended reality appliance, withoutmodifying the color scheme of a remainder of the virtual content. Asanother example, for virtual content that is meant to be all in color,the at least one processor may be configured to modify a portion of thevirtual content by changing its color scheme to a black and white orgrayscale, while leaving a remainder of the virtual content in coloruntil a target temperature is reached. In this example, the temperatureof one or more components of the wearable extended reality appliance maybe higher than a threshold temperature initially. Changing the colorscheme as described above may reduce the temperature to below thethreshold temperature. The color scheme for the virtual content mayrevert back to unmodified settings (e.g., all in color) in all portionsof the virtual content once the temperature is below the thresholdtemperature.

The particular way in which the color scheme is modified, for example,changing a portion of the virtual content to be displayed in black andwhite, may be configured by the wearable extended reality applianceuser. A user's preferred color scheme modification may be stored in theuser's profile.

In some embodiments, changing the display settings of virtual contentmay include reducing the opacity value of at least a portion of thevirtual content. Modifying opacity may refer to how transparent ortranslucent displayed virtual content appears. Virtual content displayedwith higher opacity may stand out more from the surrounding environmentand may allow less surrounding light to penetrate through it as comparedto virtual content displayed with a relatively lower opacity. Displayingcontent at maximum opacity may use more processing power, and thusgenerate more heat, than when the opacity is modified such that thecontent is more translucent or transparent.

In one example, a first portion of virtual content may be displayed at areduced opacity, and a second portion of virtual content may bedisplayed at an unreduced opacity. For example, when displayed at areduced opacity, certain graphics in a presentation may appeartranslucent to the viewer, whereas other graphics may be displayed attheir normal opacity settings. In this example, text and graphicalimages may be displayed as part of virtual content. Here, the text maybe displayed unmodified, but the graphics may be reduced in opacity toreduce heat generation, or vice versa. In this situation, thetemperature associated with one or more components of the wearableextended reality appliance may be high initially, and changing theopacity as described may reduce the temperature. Once the temperaturereduces below a predetermined threshold temperature, the opacity for thevirtual content may revert back to unmodified settings (e.g., no contentappears translucent or transparent) in all portions of the virtualcontent.

The particular way in which the opacity is modified, for example, by howmuch opacity is reduced, may be configured by the wearable extendedreality appliance user. Additionally, the particular portion of virtualcontent that is to be reduced in opacity may be configured by thewearable extended reality appliance user. A user's preferred opacitymodification may be stored in the user's profile.

In some embodiments, changing the display settings of virtual contentmay also include reducing the intensity value of at least a portion ofthe virtual content. The term intensity may refer to how much light isgenerated from the displayed virtual content via the heat-emitting lightsource. Displaying virtual content at a user's preferred intensity mayuse more processing power, and thus generate more heat, than whenintensity is reduced.

A first portion of virtual content may be displayed at a reducedintensity, and a second portion may be displayed at an unreducedintensity. For example, presented virtual content may comprise both textand graphics. The text may be displayed at unmodified settings, but whendisplayed at a reduced intensity, the graphics in the presentation mayappear less bright to the viewer because the power used to present thevirtual content has been reduced. In this example, the temperature ishigh initially, and changing the intensity as described reduces thetemperature. The intensity of the virtual content may revert back tounmodified settings (e.g., normal intensity based on a user'spreference) in all portions of the virtual content after the temperaturefalls below the threshold temperature.

The particular way in which intensity is modified, for example, by howmuch intensity is reduced, may be configured by the wearable extendedreality appliance user. Additionally, the particular portion of virtualcontent that is to be reduced in intensity may be configured by thewearable extended reality appliance user. A user's preferred intensitymodification may be stored in the user's profile.

Changing the display settings of virtual content may also includereducing a brightness value of at least a portion of the virtualcontent. The term brightness may refer to one's visual perception of howmuch light is emitted from the displayed content. While adjustingintensity of the displayed content involves adjusting how much light isgenerated by the heat-emitting light sources, adjusting brightnessinvolves adjusting the display settings of the virtual content while theintensity remains constant. Displaying virtual content at a user'spreferred brightness settings may use more processing power, and thusgenerate more heat, than when brightness is reduced.

In this embodiment, a first portion of virtual content may be displayedat a reduced brightness, and a second portion may be displayed at anunreduced brightness. For example, a presented virtual content maycontain both text and graphics. To reduce temperature, the graphics maybe displayed at a reduced brightness, but the text may be displayedunmodified, or vice versa. In this example, the temperature is highinitially, and changing the brightness as described reduces thetemperature. The brightness of the virtual content may revert back tounmodified settings (e.g., normal brightness based on a user'spreference) in all portions of the virtual content after the temperaturefalls below the threshold temperature.

The particular way in which brightness is modified, for example, by howmuch brightness is reduced, may be configured by the wearable extendedreality appliance user. Additionally, the particular portion of virtualcontent that is to be reduced in brightness may be configured by thewearable extended reality appliance user. A user's preferred brightnessmodification may be stored in the user's profile.

Changing the display settings of virtual content may also includereducing a frame rate of at least a portion of the virtual content. Aframe rate may refer to a frequency at which consecutive images aredisplayed, especially when a video and/or animated virtual objects aredisplayed. The higher the frame rate, the smoother the displayed videocontent appears to a viewer. Displaying virtual content at an unmodifiedframe rate may consume more processing power, and thus generate moreheat, than displaying virtual content at a reduced frame rate.

In this embodiment, a first portion of virtual content may be displayedat a reduced frame rate, and a second portion may be displayed at anunreduced frame rate. For example, when displayed at a reduced framerate, certain presented video content may appear choppy, i.e., themotion in the video will not be as smooth. Other portions of presentedvideo content, however, would be displayed at their normal frame rate,and will the motion will appear smooth to a viewer. In one example, theframe rate of the video may be reduced where not much is changing, suchas a scene in which there is a still landscape. Here, the frame rate inthe video may remain unmodified where the content is changing rapidly,such as characters walking down a crowded street. In this example, thetemperature is high initially, and reducing the frame rate as describedreduces the temperature. The frame rate of the virtual content mayrevert back to unmodified settings (e.g., video content appears smooth)in all portions of the virtual content after the temperature falls belowthe threshold temperature.

The particular way in which frame rate is modified, for example, by howmuch frame rate is reduced, may be configured by the wearable extendedreality appliance user. Additionally, the particular portion of virtualcontent that is to be reduced in frame rate, i.e., the beginning,middle, or end portion of a video, may be configured by the wearableextended reality appliance user. A user's preferred frame ratemodification may be stored in the user's profile.

Changing the display settings of virtual content may also includereducing a display size of at least a portion of the virtual content.Reducing the display size of virtual content may help reduce powerconsumption and thus reduces temperature. For example, larger images andvideos may have larger file sizes, which may require more power forreading from memory, processing, and/or displaying. In another example,reducing the display size may allow shutting off or turning into a lowpower mode at least part of at least one display device included in thewearable extended reality appliance.

In this embodiment, a first portion of virtual content may be displayedat a reduced display size, and a second portion may be displayed at anunreduced display size. For example, as part of displayed virtualcontent, a document may be displayed at a reduced size, whereas a videofile may be displayed at an unmodified size.

In one example, the portion of the virtual content may be selected basedon information indicative of attention of a user of the wearableextended reality appliance. In another example, the portion of thevirtual content may be selected based on degrees of necessity associatedwith different portions of the virtual content. In some embodiments theinformation indicative of attention of a user of the wearable extendedreality appliance may include captured image information. In oneexample, the information indicative of the attention of the user may bebased on a position of a cursor. In another example, the informationindicative of the attention of the user may be based on a direction ofthe wearable extended reality appliance, i.e., where the user isdirecting her head. In yet another example, the information indicativeof the attention of the user may be based a gaze direction of the user.

Additionally, the portion of virtual content to be reduced in size maybe based on degrees of necessity associated with different portions ofthe virtual content. The degrees of necessity may refer to how importantparticular content is to a particular user. For example, presentedvirtual content may have a few key points that need to be explained to aviewing user of a wearable extended reality appliance, and have othersupporting points that may not be as important. When configuring thewearable extended reality appliance, a user may assign each portion ofpresented content a value pertaining to its necessity, and the displaysize of the virtual content may be reduced based upon the assignednecessity value.

By way of example, FIG. 25 illustrates reducing the display size of aportion of the virtual content based on received temperatureinformation. Here, a wearable extended reality appliance 2510 presentsvirtual content 2512. Virtual content 2512 is comprised of virtualobjects 2514 and 2516. Upon receiving temperature data 2518 anddetermining a need to modify virtual content display settings, thedisplay size of the virtual content may be reduced. Here, virtual object2520 is reduced in size version of 2516 whereas virtual object 2522 isnot (i.e., virtual object 2522 is similar or identical in size to 2514).

In some embodiments, changing display settings of the virtual contentincludes implementing selective changes to displayed virtual objectsincluded in the virtual content based on at least one of object type orobject usage history. In the context of these embodiments, virtualobject type may refer to a characteristic of displayed content. Suchcharacteristics (or type) may include, for example, inanimate virtualcontent (e.g., a document), animate virtual content (e.g., a video orPowerPoint presentation), colored, grayscale, or black and whitecontent, transparent, translucent, or opaque content, bright or dimcontent, stationary or moving content, reduced frame rate content,and/or large display-size content and small display-size content. Objectusage history may refer to how recently the specific object was used,how frequently the object was used, how long the object was used, whattime of day the object was used, for what purpose the object was used,and/or how many users used the particular virtual object.

Display parameters that may be modified based on the type of contentpresented, i.e., a video or document, may be configured based on auser's preference. For example, a user may prefer to display a documentthat is highlighted, made brighter, or zoomed in instead of modifyingother display parameters. In another example, a user may prefer to slowdown the frame rate of presented video content in lieu of modifyingother display parameters or may prefer to automatically reduce thedisplay size of the presented virtual content.

Additionally, display parameters may be modified based on the usagehistory. For example, a wearable extended reality appliance user mayhave previously slowed down video speeds in order to reduce heatgeneration. This information may be stored in the non-transitorycomputer readable medium associated with the wearable extended realityappliance as part of the user's profile. A processor may automaticallyslow down video speeds when the wearable extended reality appliancereaches the threshold temperature based on the stored usage history inthe user's profile. In another example, the user may prefer to reducedisplay size, change the color scheme, or reduce brightness of thedisplayed content. For example, the presentation may include text andgraphics. Based on the content presented and the user's audience, theuser may prefer to reduce the display size of graphics, change thegraphics to grayscale, or reduce the brightness of the graphics in orderto reduce temperature, but without changing the content and meaning ofthe presentation. Such display parameter modification information may bestored in the non-transitory computer readable medium so that thewearable extended reality appliance may reduce heat generation based onusage history. In another example, a particular object that has not beenused for a lengthy period of time may be reduced in intensity,brightness, or any other working parameter. For example, a virtualobject, such as a document as part of a presentation, that has not beenused for two hours may be reduced in intensity.

In some embodiments, changing the display settings of the virtualcontent may include removing from the virtual content at least onevirtual element of a plurality of virtual elements included in thevirtual content. In the context of some disclosed embodiments, a virtualelement may be a virtual object or part of a virtual object. Forexample, if virtual content is comprised of multiple documents, avirtual object may be one of the documents. A virtual element may be thevirtual object itself, e.g., the document, or may be part of the virtualobject, such as the task bar, navigation pane, or styles tab. To reduceheat generation, a virtual element, such as a task bar or navigationpane, may be removed from the displayed virtual content. For example, apresented virtual content may involve a document. To reduce heatgeneration, the navigation pane of the document may be removed, i.e., itis not visible to the viewer.

In some embodiments, the at least one virtual element may be selectedfrom the plurality of virtual elements based on information indicativeof attention of a user of the wearable extended reality appliance. Theattention of the user may be used to remove a virtual element becausesuch data is indicative of what portions of the virtual content arecrucial to conveying the meaning of the virtual content, and which arenot. Information indicative of the attention of the user may be based onimage data captured by the image sensor associated with the wearableextended reality appliance. In one example, the information indicativeof the attention of the user may be based on a position of a cursor. Inanother example, the information indicative of the attention of the usermay be based on a direction of the wearable extended reality appliance.In this example, a user may need to move his or head or gaze tocompletely view virtual content. The image sensor may capture datarelated to where a user's attention is, i.e., where their head islocated, and modify the virtual content accordingly. In yet anotherexample, the information indicative of the attention of the user may bebased a gaze direction of the user. In another example, the image sensormay capture gesture data. Virtual objects to which the user is pointingto may be displayed at unmodified settings, whereas virtual objects thatthe user does not point to may be displayed at reduced brightness,intensity, frame rate, opacity, or at a modified color.

Some disclosed embodiments may include ranking importance level of theplurality of virtual elements, and the at least one virtual element isselected from the plurality of virtual elements based on the determinedlevel of importance. The importance level may refer to how necessarycertain virtual elements are for the user to understand the presentedvirtual content. For example, an image may be presented as part ofvirtual content, but may not be necessary to understand the rest of thevirtual content. The image may be assigned an importance level that islower than the remainder of the presented virtual content. In anotherexample, a presentation may be presented as virtual content (forexample, in a presentation editing and/or viewing application) and mayinclude slides and notes. The notes may not be as important as theslides and may therefore be assigned a lower importance level comparedto the slides. By way of another example, if the virtual element is avirtual productivity application, information directly related to usingthe productivity application, such as documents, emails, or explanatoryvideos, may be assigned a high level of importance, and thus it would bea priority to display such content unmodified. In contrast, virtualelements that are not directly related to using the productivityapplication, such as videos playing music, open web pages, or otherapplications unrelated to the productivity application (or to corefunctions of the productivity application), may be assigned a lowerlevel of importance. These elements may be modified in order to reduceheat generation. Additionally, if a virtual element is part of thevirtual window, components that are not being used at the moment, suchas a task bar, navigation pane, or styles tab, may be removed.

In some examples, ranking importance level of the plurality of virtualelements may comprise ranking importance level of the plurality ofvirtual elements based on a condition of the physical environment of thewearable extended reality appliance and/or an event in the physicalenvironment of the wearable extended reality appliance. For example,when another person other than the user of the wearable extended realityappliance is present in the physical environment of the wearableextended reality appliance, virtual elements associated with the otherperson may be ranked higher than when the other person is not present inthe physical environment of the wearable extended reality appliance. Inanother example, when an electrical appliance is present in the physicalenvironment of the wearable extended reality appliance, controlsassociated with the electrical appliance may be ranked higher than whenthe electrical appliance is not present in the physical environment ofthe wearable extended reality appliance. In yet another example, when aperson is physically approaching a user of the wearable extended realityappliance, elements associating with content sharing may be rankedhigher than when no person is not physically approaching the user. Insome examples, determining the present of a person and/or of anelectrical appliance in the physical environment of the wearableextended reality appliance, and/or determining whether a person isphysically approaching the user, may be determined based on an analysisof an image data captured using an image sensor included in the wearableextended reality appliance. The analysis may include analyzing the imagedata using a visual classification algorithm to one class of a pluralityof alternative classes (such as ‘other person present’, ‘no other personpresent’, ‘electrical appliance present’, ‘no electrical appliancepresent’, ‘person approaching the user’, ‘no person approaching theuser’, or any other combination of additional users and/or appliances).

Some disclosed embodiments may include receiving, over a period of timefollowing implementation of the change to the display settings, updatedinformation indicative of the temperature associated with the wearableextended reality appliance and altering at least one of the displaysettings to an initial value. As described above, temperature values maybe received periodically, continuously, or based on set temperatureincreases (i.e., no information is generated unless the temperatureincreases one or two degrees Celsius). How the processor receivestemperature values may be configured by the wearable extended realityappliance user. Thus, temperature values may be received continuously orperiodically during use of the wearable extended reality appliance. Forexample, temperature measurements by a temperature sensor associatedwith the wearable extended reality appliance may indicate that thetemperature of one or more components of the wearable extended realityappliance has decreased below a threshold temperature, or have decreasedto a safe or stable temperature. In one example, in response, thewearable extended reality appliance may be used without anymodifications to display settings. In another example, in response, atleast one of the display settings may be altered to an initial value. Insome embodiments, the initial value may refer to a display settingbefore being changed (for example, altering display settings to theinitial value 2418 in FIG. 24).

In the above example, the working parameter (such as voltage, current,or power) may be remodulated by the at least one processor to an initialvalue after the wearable extended reality appliance has not been used.Accordingly, the initial value may refer to a display setting or to aworking parameter before being changed. One example of an initial valuemay be a default display setting. In this example the initial value mayrefer to a default color, opacity, intensity, brightness, frame rate, ordisplay size setting. The initial value of a display setting may beconfigured by each individual wearable extended reality appliance user.

FIG. 26 is a flow chart illustrating an exemplary method for hangingdisplay settings based on a temperature of a wearable extended realityappliance. Method 2600 may be performed by one or more processingdevices (e.g., 360, 460, or 560) associated with input unit 202 (seeFIG. 3), XR unit 204 (see FIG. 4), and/or remote processing unit 208(see FIG. 5). The steps of the disclosed method 2600 may be modified inany manner, including by reordering steps and/or inserting or deletingsteps. Method 2600 may include a step 2612 of displaying virtual contentvia a wearable extended reality appliance. One or more components of thewearable extended reality appliance (e.g., 412, 413, 414, 415, 417) (seeFIG. 4) may generate heat while displaying the virtual content. Method2600 may include a step 2614 of receiving temperature informationassociated with a wearable extended reality appliance based on the heatgeneration (for example, after or while step 2612 displays the virtualcontent). Method 2600 may include a step 2616 of determining a need tochange display settings based on received temperature information. Theat least one processing device (e.g., 360, 460, 560) associated withinput unit 202 (see FIG. 3) may determine the need to change displaysettings based on received temperature information. Method 2600 mayinclude a step 2618 of based on the determination of step 2616, changingthe display settings of the virtual content to achieve a targettemperature. The display settings may be changed to reach the targettemperature.

Some disclosed embodiments may include systems, methods, and/ornon-transitory computer readable media containing instructions that whenexecuted by at least one processor may cause the at least one processorto perform operations for implementing hybrid virtual keys in anextended reality environment. An extended reality environment maycontain real and virtual elements. A user may interact with the real andvirtual elements while operating in the extended reality environment. Insome embodiments, the extended reality environment may contain hybridvirtual keys. Hybrid virtual keys may include a mix of physical andvirtual keys. Physical keys may represent one or more physical and/ormechanically movable keys of, for example, a keyboard. A keyboard mayallow a user to input text and/or alphanumeric characters using, forexample, one or more keys associated with the keyboard. Virtual keys mayallow the input of text and/or alphanumeric characters and/or specialpurpose keys (such as command, control and/or function keys) without theneed for physical keys. Virtual keys may have the same appearance asphysical keys, or a different appearance, but may not have a physical,movable component. A user may interact with virtual keys through atouchscreen interface or in the extended reality environment. In someembodiments, the hybrid keys may include physical keys and virtual keys.For example, the extended reality environment may contain a physicalkeyboard with physical keys and a surface including virtual keys. Insome embodiments, the hybrid virtual keys may correspond to number keys,letter keys, symbol keys, up/down arrows, special purpose keys (such ascommand, control and/or function keys), or other types of keys.

Some disclosed embodiments may include receiving, during a first timeperiod, first signals corresponding to positions on a touch-sensitivesurface of a plurality of virtual activatable elements virtuallyprojected by a wearable extended reality appliance on thetouch-sensitive surface. A touch-sensitive surface may include a surfacethat may generate signals in response to being touched by an object. Thetouch-sensitive surface may be located on one or more physical objectspresent in an environment of the user. For example, the touch-sensitivesurface may include one or more surfaces of a physical keyboard. As oneexample, a touch bar or a touch pad on a physical keyboard mayconstitute a touch-sensitive surface. It is contemplated, however, thatother portions of the keyboard, such as, sides or one or more physicalkeys of the keyboard may be a touch-sensitive surface. As anotherexample, the touch-sensitive surface may include a surface of a table,desk, or any other objects in the environment of the user. An objectused to touch the touch-sensitive surface may include one or moreportions of a user's hand, a stylus, a pointer, or any other physicalobject that may be brought into contact with the touch-sensitivesurface. As one example, a user may use one or more fingers, thumbs,wrist, palm, or any other portion of the user's hand to touch thetouch-sensitive surface.

In some embodiments, the touch-sensitive surface may include one or morevirtual activatable elements. Virtual activatable elements may includeone or more widgets that may be displayed or projected onto thetouch-sensitive surface. For example, virtual activatable elements mayinclude icons, symbols, buttons, dials, check boxes, selection boxes,pull-down menus, sliders, or any other type of graphical element thatmay be displayed or projected onto the touch-sensitive surface. Eachvirtual activatable element may generate a signal or cause an action tobe taken when operated or acted on by a user. For example, a user mayactivate a virtual activatable button element by pressing on or touchingthe button. As another example, a user may activate a virtualactivatable slider element by using a finger to swipe upwards ordownwards on the element. A virtual activatable element may generate asignal or cause a processor to take an action in response to beingtouched or activated by a user.

A virtual activatable element displayed or projected onto atouch-sensitive surface may be associated with a position. The positionmay be associated with a location of a respective virtual activatableelement relative to a reference location or a reference surface. In someembodiments, the position may be determined using coordinate elements.For example, a position of a virtual activatable element may includecoordinate locations or distances relative to a predeterminedco-ordinate reference plane. As another example, a position of a virtualactivatable element may include a condition of a button (e.g., on oroff) on a touch-sensitive surface.

In some embodiments, the virtual activatable elements may be virtuallyprojected by a wearable extended reality appliance. This projection maybe seen on the touch-sensitive surface by a user wearing the wearableextended reality appliance. Further, this projection may be invisible toother people, or may be seen away of the touch-sensitive surface byother people (for example, people not using wearable extended realityappliances, people not using the particular wearable extended realityappliance, and so forth). That is, a person looking at thetouch-sensitive surface without a wearable extended reality appliancewould not see the virtual activatable elements on the touch-sensitivesurface. A wearable extended reality appliance may be a device that isattachable to a user to provide an augmented reality (AR), virtualreality (VR), mixed reality (MR), or any immersive experience. Typicalcomponents of wearable extended reality appliances may include at leastone of: a stereoscopic head-mounted display, a stereoscopic head-mountedsound system, head-motion tracking sensors (such as gyroscopes,accelerometers, magnetometers, image sensors, structured light sensors,etc.), head mounted projectors, eye-tracking sensors, and/or additionalcomponents described below. The wearable extended reality appliance mayinclude smart glasses, headsets, or other appliances. The wearableextended reality appliance may virtually project the virtual activatableelements by providing an overlay of the virtual activatable elements onthe touch-sensitive surface such that the virtual activatable elementsmay be displayed to a user using the wearable extended reality device.

By way of example, FIG. 27 illustrates an example of a wearable extendedreality appliance virtually projecting one or more virtual activatableelements onto a touch-sensitive surface. For example, as illustrated inFIG. 27, user 2716 may be wearing wearable extended reality appliance2715. In some examples, wearable extended reality appliance 2715 mayinclude smart glasses, a headset, a head-mounted display, or any otherform of wearable extended reality appliance. Wearable extended realityappliance 2715 may virtually project one or more virtually activatableelements 2712, 2713, and/or 2714 onto touch-sensitive surface 2711 thatmay be located below keyboard 2710. The projected one or more virtuallyactivatable elements may be visible to user 2716. In one example, theprojected one or more virtually activatable elements may be invisible toother people. In another example, the projected one or more virtuallyactivatable elements may be seen away of touch-sensitive surface 2711 byother people (for example, as a reflection over wearable extendedreality appliance 2715). That is, a person looking at thetouch-sensitive surface without a wearable extended reality appliancewould not see the virtual activatable elements on the touch-sensitivesurface.

In some embodiments, a processor may receive first signals correspondingto positions on a touch-sensitive surface during a first time period.The first time period may be a length of time during which one or moreof actions may takes place. The length of time may be variable. Forexample, the first time period may be a few milliseconds, seconds,minutes, or any other duration of time. The first signals received bythe processor may correspond to positions on a touch-sensitive surface.The first signals may represent a specific position of a virtualactivatable element. The processor may receive signals indicatinglocations of each of the projected virtual activatable elements when thewearable extended reality appliance displays the virtual activatableelements, before the wearable extended reality appliance displays thevirtual activatable elements, or after the wearable extended realityappliance displays the virtual activatable elements. For example, awearable extended reality appliance may project a brightness adjusterscroll bar onto a touch-sensitive surface. A processor may receive afirst signal indicating the brightness adjuster scroll bar is located ona right side of the touch-sensitive surface.

By way of example, FIG. 28 illustrates an example of a keyboard and atouch-sensitive surface. As illustrated in FIG. 28, touch-sensitivesurface 2811 may be located below keyboard 2810 but may additionally oralternatively be located elsewhere. For example, touch-sensitive surface2811 may be located above keyboard 2810, to the left or right ofkeyboard 5-2-10, or be in-between physical keys of keyboard 2810. Insome embodiments, touch-sensitive surface 2811 may also be located onkeyboard 2810 itself. Touch-sensitive surface 2811 may include, forexample, areas corresponding to one or more virtual activatableelements. For example, as illustrated in FIG. 28, the virtualactivatable elements may include arrows 2814, a brightness adjustment2813, and/or one or more icons representing one or more applications2812. It is to be understood that touch-sensitive surface 2811 is notlimited to these illustrated virtual activatable elements and mayinclude any of several other areas corresponding to such elements asdescribed above.

Some disclosed embodiments may include arranging the plurality ofvirtual activatable elements on the touch-sensitive surface based on adefault arrangement previously selected by the user. Prior to performingone or more operations in an extended reality environment, a user mayspecify one or more locations and/or arrangements of one or more virtualactivatable elements as a default arrangement. A processor may use theuser-specified locations to cause the virtual activatable elements to bedisplayed in the default arrangement specified by the user. For example,a user may choose to arrange all of the virtual activatable elements onone side of the surface. In another example, a user may group certainvirtual activatable elements together. In another example, a user mayarrange the elements with the same distance between each element. By wayof another example, a user may arrange the elements with differentdistances between each other. A user may also modify a previouslyspecified default arrangement. For example, the user may drag one ormore virtual activatable elements to one or more new locations. In someembodiments, the user may push a key to indicate a change in location ofa virtual activatable element. For example, a user may push a down arrowkey to indicate that the location of the virtual activatable elementshould be moved in a downward direction. In response to one or more userinputs, a processor may rearrange the positions of one or more of thevirtual activatable elements. In some examples, default arrangementpreviously selected by the user may be context depended. For example,the default arrangement may specify different arrangements for differentworking modes of the wearable extended reality appliance (for example,one arrangement for power-saving mode and another arrangement fornon-power-saving mode, one arrangement for lay-back usage mode of thewearable extended reality appliance and another arrangement fornon-lay-back usage mode, one arrangement when the device is used whilewalking or running and a different arrangement when the device is usedwhile standing or sitting, and so forth). In another example, thedefault arrangement may specify different arrangements for differentvirtual content consumed using the wearable extended reality appliance(for example, one arrangement when using a word processer and anotherarrangement when viewing a movie, one arrangement when in videoconference and another arrangement when surfing the web, one arrangementwhen the wearable extended reality appliance is used to share contentwith other wearable extended reality appliances and another arrangementotherwise, and so forth). In yet another example, the defaultarrangement may specify different arrangements for different physicalenvironment conditions. For example, the default arrangement may specifyone arrangement for when a user of the wearable extended realityappliance is approached by another person and a different arrangementotherwise, one arrangement when the wearable extended reality applianceis used in a meeting room and another arrangement when the wearableextended reality appliance is used in a home settings, and so forth.

In some disclosed embodiments the plurality of virtual activatableelements virtually projected on the touch-sensitive surface may be asub-set of a group of virtual activatable elements, and the sub-set maybe determined based on an action of the user. The sub-set may be aproper sub-set of the group of virtual activatable elements (that is, atleast one virtual activatable element of the group of virtualactivatable elements is not included the sub-set). Further, the sub-setmay be a non-empty sub-set (that is, at least one virtual activatableelement of the group of virtual activatable elements is included in thesub-set). Some non-limiting examples of such action of the user mayinclude usage of a particular application, using a particular functionof a particular application, switching to a lay-back usage mode,switching to an airplane mode, pairing the wearable extended realityappliance with a particular device, walking, running, standing, sitting,communicating with a person in a physical vicinity of the user, enteringa specific physical space, and so forth. In one example, the action ofthe user may be detected, for example by analyzing data received fromone or more sensors and/or based on inputs received from the user, forexample as described herein. Further, upon a detection of the action,the sub-set of the group of virtual activatable elements may selected.One or more of the virtual activatable elements may be grouped togetherin a sub-set of all virtual activatable elements being presented to auser. In some embodiments, the one or more virtual activatable elementsmay be grouped based on functions performed by those elements. Forexample, virtual activatable elements related to editing text orchanging display parameters may be grouped together. In someembodiments, the one or more virtual activatable elements may be groupedbased specific applications. For example, virtual activatable elementsrelated to a presentation editing and/or viewing application may begrouped together. As another example, virtual activatable elementsrelated to a video editing or playback application may be groupedtogether. In some embodiments, one or more of the virtual activatableelements may be grouped together in a sub-set based on an action takenby a user. For example, an action taken by the user may include editinga word document. The sub-set of virtual activatable elements may includeoptions for a user to remove or add words or to change the font sizeand/or style. As another example, an action may include a user editing avideo file. The sub-set of elements may include options for a user todelete or add audio or video clips.

In some disclosed embodiments the plurality of virtual activatableelements virtually projected on the touch-sensitive surface may be asub-set of a group of virtual activatable elements, and the sub-set maybe determined based on a physical location of the user. The sub-set maybe a proper sub-set of the group of virtual activatable elements (thatis, at least one virtual activatable element of the group of virtualactivatable elements is not included the sub-set. Further, the sub-setmay be a non-empty sub-set (that is, at least one virtual activatableelement of the group of virtual activatable elements is included in thesub-set. A user may perform different types of actions in differentsettings. For example, different virtually activatable elements may beused when a user is in a home, office, conference room, in public, whiletravelling, or when the user is in other settings. The processor maycontain a GPS device or an indoor positioning sensor to determine alocation of the user. The processor may display a sub-set of virtualactivatable elements based on the determined location. For example, whena user is physically located in his home, the processor may display asub-set of virtual activatable elements associated with an emailapplication. As another example, when a user is physically located inhis workplace, the processor may display sub-sets of virtual activatableelements associated with document editing and/or video editing elements.In another embodiment, the sub-set of virtual activatable elements maybe determined based on a physical location of at least one of the user,the wearable extended reality appliance, or the touch-sensitive surface.For example, a GPS device may be attached to the wearable extendedreality appliance to determine the location of the appliance. Differentsub-sets of virtual activatable elements may be projected by thewearable extended reality appliance on a touch-sensitive surface basedon the location of the appliance. For example, a wearable extendedreality may be located in a user's office. A sub-set of virtualactivatable elements associated with document sharing may be projectedbased on the location. As another example, a GPS device may be attachedto the touch-sensitive surface to determine a location of that surface.Different sub-sets of virtual activatable elements may be projectedbased on the location of the surface. For example, a touch-sensitivesurface may be located in a user's home. A sub-set of virtualactivatable elements associated with display changes may be projectedbased on the location. In the examples discussed above, the sub-sets maybe proper subsets.

In some disclosed embodiments the plurality of virtual activatableelements virtually projected on the touch-sensitive surface may be asub-set of a group of virtual activatable elements, and wherein thesub-set may be determined based on an event in an environment of theuser. The sub-set may be a proper sub-set of the group of virtualactivatable elements (that is, at least one virtual activatable elementof the group of virtual activatable elements is not included thesub-set. Further, the sub-set may be a non-empty sub-set (that is, atleast one virtual activatable element of the group of virtualactivatable elements is included in the sub-set. The sub-set of virtualactivatable elements may be determined based on a specific action takenby the user or a change in the user's physical or virtual environment.In some embodiments, the specific action may be an event that does notinvolve the user. For example, an action that may not involve the usermay include another user connecting to the same extended realityenvironment to which the user may be connected. By way of example, whenanother user connects to the extended reality environment of the user,the virtual activatable elements may change to editing elements so thatboth users can perform editing functions. In another example, an actionthat may not involve the user may include another person physicallyapproaching the user and/or physically entering a physical environmentof the user. In some embodiments, a specific action that may determinethe virtual activatable elements included in a sub-set may include theentrance of another person. For example, the entrance of another personmay cause some virtual activatable elements to disappear from thetouch-sensitive surface because the other person may not be authorizedto perform functions associated with those virtual activatable elements.

In some embodiments, the event that determines the sub-set of virtualactivatable elements may occur in a physical environment associated witha user. For example, entrance of another person into the user's home maybe an event that may determine the sub-set of virtual activatableelements that are displayed to the user. As another example, an eventoccurring in the physical environment may involve a change intemperature in a user's workplace. In some embodiments, the event thatdetermines the sub-set of virtual activatable elements may occur in avirtual environment. For example, the event may include a notificationreceived by a user in the user's extended reality environment. A sub-setof virtual activatable elements may be selected or modified based on thenotification. For example, if the notification is an email, the sub-setof virtual activatable elements may change to functions associated withan email application. As another example, if the notification is a phonecall, the sub-set of virtual activatable elements may change tofunctions associated with a phone.

Some disclosed embodiments may include determining from the firstsignals the positions of the plurality of virtual activatable elementson the touch-sensitive surface. When a virtual activatable element isactivated, actuated, projected by the wearable extended realityappliance, or triggered, one or more signals (such as the first signals)may be generated. A processor may receive the one or more signals anddetermine a position of the virtual activatable element based on the oneor more signals. In some embodiments, the signal may include anindication of a position of the virtual activatable element. In oneembodiment, the signal may store the position of the virtual activatableelement based on the user-determined position. The signal may includeposition information in the form of coordinates or distances from acoordinate plane. The processer may receive the position information andmay determine the position of the virtual activatable element based onthe information. For example, a processor may determine the virtualactivatable element is located on the left of the touch-sensitivesurface, or at any position to the left of the touch-sensitive surface,based on position information received with the signal. In anotherembodiment, the position of each virtual activatable element may beassociated with the virtual activatable element and stored in a datastructure. The signal may include an identification of the virtualactivatable element that may be used as an index to search the datastructure to determine the position. As an example, the processor maydetermine the virtual activatable element is located on the right of thetouch-sensitive surface, or at any location to the right of thetouch-sensitive surface, based on an identification received with thesignal. As another example, the processor may determine the virtualactivatable element may be a slider and the position may be at eitherend of the slider, or at any position between the two ends. As anotherexample, the virtual activatable element may be a dial and the positionmay be at any rotational angle along the dial.

In some disclosed embodiments, the positions of the plurality of virtualactivatable elements on the touch-sensitive surface may be determinedbased on at least one of an action of the user, a physical location ofthe user, a physical location of the wearable extended realityappliance, a physical location of the touch-sensitive surface, or anevent in an environment of the user. Different criteria may be used todetermine positions (or layout) of virtual activatable elementsdisplayed to a user. The positions of one or more of the virtualactivatable elements may also be altered based on one or more criteria.For example, when a user is in an office environment, positions fordisplaying the virtual activatable elements may be selected so that theelements appear on more than one area. As another example, when a useris in a home environment, positions for displaying the virtualactivatable elements may be selected so that the elements appear on asingle area. As another example, different positions for displaying thevirtual activatable elements may be determined based on an action takenby the user or an action taken by someone who is not the user. Forexample, a user may open an application and the positions of the virtualactivatable elements may move to the left or right to create space fordisplaying the opened application. In some other examples, the positionsof virtual activatable elements associated with a first application maybe selected based on whether other applications are used. For example,when the first application is used alone, the virtual activatableelements associated with the first application may be spread across theentire touch-sensitive surface, and when the first application is usedin conjunction with one or more other applications, the virtualactivatable elements associated with a first application may bepositioned in a selected portion of the touch-sensitive surface, whilevirtual activatable elements associated with the one or more otherapplications may be positioned on the touch-sensitive surface outsidethe selected portion. The portion of the touch-sensitive surface may beselected based on the first application, based on the one or more otherapplications, and so forth.

Some disclosed embodiments may include receiving a touch input of a uservia the touch-sensitive surface, wherein the touch input includes secondsignals generated as a result of interactions with at least one sensorwithin the touch-sensitive surface. A touch-sensitive surface maycontain one or more sensors to generate signals. The one or more sensorsmay detect or measure events or changes in an environment or may detectwhether a user is touching a touch-sensitive surface. The one or moresensors may include image sensors, position sensors, pressure sensors,temperature sensors, or any other sensor capable of detecting one ormore characteristics associated with the touch-sensitive surface or withan environment in which the touch-sensitive surface is located. The usermay touch the surface by engaging with the touch-sensitive surface. Forexample, a user may apply pressure to part of the touch-sensitivesurface. As another example, a user may tap, touch, press, brush, orflick part of a touch-sensitive surface. For example, the user may pressthe touch-sensitive surface that corresponds to a letter key. The user'sengagement with the touch-sensitive surface may cause one or moresensors associated with the touch-sensitive surface to generate signals.For example, a sensor associated with the touch-sensitive surface maysend a signal to the processor that a particular letter key has beentouched by the user.

By way of example, FIG. 29 illustrates an example of a user interactingwith touch-sensitive surface. For example, as illustrated in FIG. 29,user's hand 2915 may touch an area of touch-sensitive surface 2911 onwhich virtual activatable element 2912 is projected. Touch-sensitivesurface 2911 may be associated with keyboard 2910. User's hand 2915 maytouch the area of touch-sensitive surface 2911 corresponding to virtualactivatable element 2912 and a sensor may generate a signal reflectingthat. The processor may receive the signal and open an applicationassociated with virtual activatable element 2912. In another example,user's hand 2915 may touch an area of touch-sensitive surface 2911 onwhich virtual activatable element 2913 is projected and a sensor maygenerate a signal reflecting that. The processor may receive the signaland perform an action associated with virtual activatable element 2913.In another example, user's hand 2915 may touch an area oftouch-sensitive surface 2911 on which virtual activatable element 2914is projected, and a sensor may generate a signal reflecting that. Theprocessor may receive the signal and perform an action associated withvirtual activatable element 2914.

Some disclosed embodiments may include determining a coordinate locationassociated with the touch input based on the second signals generated asthe result of the interactions with the at least one sensor within thetouch-sensitive surface. As discussed above, when a user touches atouch-sensitive surface, one or more signals (e.g., second signals) maybe generated by one or more sensors associated with the touch-sensitivesurface. The one or more second signals may include informationassociated with a coordinate location of the touch input. A coordinatelocation may designate the position of a point relative to a givenreference frame. In some embodiments, the coordinate location may begiven as latitude and longitude. In other embodiments, a coordinatesystem (e.g., Cartesian coordinate system, polar coordinate system) maybe used to determine the coordinate location. In other embodiments, anumber line may be used to determine the coordinate location. The numberline may include a straight horizontal line with numbers placed at evenincrements along the line. The numbers may correspond to a location of atouch input. For example, the user may touch a sensor in the middle ofthe touch-sensitive surface, which may be associated with a number 10 ofa number line that may include 20 numbers. The at least one sensor maygenerate signals and the processor may determine that the touch inputwas in the middle of the touch-sensitive surface based on the signalsand the number. As another example, the user may touch a sensor on theleft or right of the touch-sensitive surface. The at least one sensormay generate signals and the processor may determine that the touchinput was on the left or right of the touch-sensitive surface based onthe signals.

Some disclosed embodiments may include comparing the coordinate locationof the touch input with at least one of the determined positions toidentify one of the plurality of virtual activatable elementscorresponding to the touch input. The processor may compare a touchinput of a user to a pre-determined position to determine which virtualactivatable elements the user may be trying to trigger. The processormay compare the coordinate location to the determined position todetermine if the coordinate location and determined position are thesame. For example, the coordinate location of the touch input may be inthe center of the touch-sensitive surface. The determined position of aslider, for example, may also be in the center of the touch-sensitivesurface. The processor may compare the two locations to determine thatthe touch input corresponds to the user touching the slider.

Some disclosed embodiments may include causing a change in a virtualcontent associated with the wearable extended reality appliance, whereinthe change corresponds to the identified one of the plurality of virtualactivatable elements. Virtual content being displayed to a user mayinclude one or more items displayed to the user using a wearableextended reality appliance. For example, virtual content may includevirtual display screens (also referred to as virtual displays herein),widgets, documents, media items, photos, videos, virtual characters,virtual objects, extended reality environment, virtual activatableelements, and other graphical or textual content. In some embodiments, aprocessor may cause a change in virtual content by adjusting one or moreitems displayed to a user. For example, the processor may adjust thesize of one or more virtual display screens. As another example, theprocessor may add one or more additional virtual display screens orremove one or more of the virtual display screens. As another example, aprocessor may play audio or display pictures to a user. The change invirtual content may be based on a virtual activatable element touched bythe user and identified by the processor by comparing the coordinatelocation to the determined position. For example, a user may touch avirtual activatable element that corresponds to a zoom in feature. Inresponse, the processor may change the virtual content by zooming in onthe content. As another example, a user may touch a virtual activatableelement that corresponds to a delete feature. In response, the processormay change the virtual content by removing an item displayed to theuser. In another example, a user may touch a virtual activatable elementthat corresponds a particular application, and in response the processormay activate the particular application. In yet another example, a usermay touch a virtual activatable element that corresponds to a particularfunction in a particular application, and in response the particularfunction may be triggered. In some examples, in response to a firstidentified one of the plurality of virtual activatable elements, a firstchange in the virtual content associated with the wearable extendedreality appliance may be caused, and in response to a second identifiedone of the plurality of virtual activatable elements, a second change inthe virtual content associated with the wearable extended realityappliance may be caused, the second change may differ from the firstchange. In some examples, in response to a first identified one of theplurality of virtual activatable elements, a first change in the virtualcontent associated with the wearable extended reality appliance may becaused, and in response to a second identified one of the plurality ofvirtual activatable elements, causing the first change in the virtualcontent associated with the wearable extended reality appliance may beavoided. In some examples, a data-structure associating virtualactivatable elements with different alternative changes to the virtualcontent associated with the wearable extended reality appliance may beaccessed based on the identified one of the plurality of virtualactivatable elements to select the change in the virtual contentassociated with the wearable extended reality appliance.

Some disclosed embodiments may include determining a type of the touchinput based on the second signals, and wherein the change in the virtualcontent corresponds to the identified one of the plurality of virtualactivatable elements and the determined type of the touch input. Thetouch input may include different types of gestures performed by theuser to touch the virtual activatable elements. A processor maydetermine the type of gesture associated with a touch input and relatethe type of gesture to a specific function. The processor may alsochange the virtual content by performing the function of the virtualactivatable element. In some embodiments, the determined type of thetouch input may include, for example, a tap, a long touch, amulti-touch, a drag touch, a flick touch, a pinch-in touch, a pinch-outtouch, a swipe touch, or a hovering touch. The touch input may beperformed by a user's hand or part of a user's hand. For example, thetouch input may be performed by a user's finger, fingertip, palm, orwrist. In some embodiments, the determined type of the touch input(e.g., gesture) may cause a change in virtual content. For example, apinch-in type touch input may correspond to a zoom in feature. Inresponse, the processor may cause the display to be magnified to zoom inon the virtual content. In another example, a drag type touch input maycorrespond to moving a virtual activatable element. In response, theprocessor may move the identified virtual activatable element based onthe drag touch input.

In some disclosed embodiments the virtual content may include a virtualdisplay, and the touch-sensitive surface may be located adjacent atouchpad configured for use in navigating a cursor in the virtualdisplay. A touchpad and touch-sensitive surface may be located next toeach other in an extended reality environment. A touchpad may include anarea that may be used for navigating a cursor. A cursor may include anindicator that appears in the virtual display to display a selectedposition on the display. The cursor may identify a point on the virtualdisplay that may be affected by a user input. For example, a user inputmay include a user interacting with the touchpad to move the cursor inthe virtual display. As a user moves his hand, or part of his hand, onthe touchpad, the cursor may move in the same manner in the virtualdisplay. For example, the user may move his fingers to the left acrossthe touchpad. In response, the cursor may move left in the virtualdisplay. As another example, the user may push down on the touchpadwhile the cursor is hovering over an application. In response, thecursor may select an application being pointed at by the cursor based onthe user pushing down on the touchpad.

In some disclosed embodiments the virtual content may include a virtualdisplay, and the operations may further include enabling thetouch-sensitive surface to navigate a cursor in the virtual display. Thetouch-sensitive surface may include an area that may be used forcontrolling a cursor. A user may interact with the touch-sensitivesurface using the user's hand or part of a user's hand, such as theuser's fingers, palm, or wrist. For example, a user may drag his handover the touch-sensitive surface, causing the cursor to move in thevirtual display. By way of another example, a user may move his fingersto the right across the touch-sensitive surface. In response, the cursormay move right in the virtual display. As another example, the user maypush down on the touch-sensitive surface while the cursor is hoveringover an application. In response, the cursor may select an applicationbeing pointed at by the cursor based on the user pushing down on thetouch-sensitive surface.

By way of example, FIG. 30 illustrates an example of a user interactingwith the touch-sensitive surface to navigate a cursor. For example, asillustrated in FIG. 30, user's hand 3015 may engage with touch-sensitivesurface 3011, which may be located below keyboard 3010. Touch-sensitivesurface 3011 may include areas corresponding to one or more virtualactivatable elements 3012, 3013, and 3014. User's hand 3015 may engagewith touch-sensitive surface 3011 to navigate cursor 3017 in virtualdisplay 3016. For example, moving hand 3015 towards the right whiletouching touch-sensitive surface 3011 may cause cursor 3017 in virtualdisplay 3016 to move towards the right.

Some disclosed embodiments may include upon detection of the touchinput, opening an application and wherein causing the change in thevirtual content is based on the opening of the application. A processormay receive a touch input from a user. The processor may compare thecoordinate location of the touch input to determined positions of theone or more virtual activatable elements and determine that the inputcorresponds to a virtual activatable element associated with anapplication. The application may include, for example, a word processor,web browser, presentation software, video software, spreadsheetsoftware, or any other type of application that may allow a user toperform some operations. The processor may open an applicationassociated with a virtual activatable element based on a touch inputassociated with that virtual activatable element. Opening of theapplication may cause a change in the virtual content by altering whatis seen by the user in the extended reality environment. For example,the touch input may correspond to a word application. The change invirtual content may include adjusting the virtual content to include theopened word document. In another example, the touch input may correspondto a presentation application. The change in virtual content may includeadjusting the virtual content to include the opened presentation.

Some disclosed embodiments may include upon detection of the touchinput, changing an output parameter; and wherein causing the change inthe virtual content is based on the change in the output parameter. Anoutput parameter may be a characteristic or feature of the virtualcontent. For example, an output parameter may include contrast,illumination, distance, sizing, volume, brightness, and/or one or moreother parameters that affect how virtual content is displayed (forexample, on a virtual display, in an extended reality environment, usingthe wearable extended reality appliance, and so forth). The processormay change an output parameter based on the determined position of thevirtual activatable element. For example, the processor may increase ordecrease a value or level associated with the output parameter based onthe determined position of the virtual activatable element. An increaseor decrease in a value or level associated with the output parameter maycause a change in the virtual content by adjusting what is seen by theuser in the extended reality environment. For example, a change in anoutput parameter such as contrast may adjust a contrast of the extendedreality display. In another example, a change in an output parameter maycause the processor to change the size of one or more windows in thevirtual content. In some embodiments, the touch input may include theuser adjusting a virtual activatable element that corresponds to ascroll bar for adjusting brightness. For example, the touch input mayinclude the user dragging the scroll bar to the left to decreasebrightness. In another example, the touch input may be the user draggingthe scroll bar to the right to increase brightness. In response, theprocessor may cause a brightness of the extended reality display toincrease or decrease based on the touch input.

Some disclosed embodiments may include deactivating at least onefunctionality of at least a portion of the touch-sensitive surfaceduring a second time period when the plurality of virtual activatableelements are not projected by the wearable extended reality applianceonto the touch-sensitive surface. When the one or more virtualactivatable elements are not projected by the wearable extended realityappliance onto the touch-sensitive surface, some or all portions of thetouch-sensitive surface may be turned off by deactivating thefunctionality. When a portion of the touch-sensitive surface is turnedoff, even though a user may touch that portion of the touch-sensitivesurface, a touch input will not be received, and the virtual contentwill not change. In some embodiments, such deactivation may occur duringa second time period. The second time period may be a length of timeduring which the one or more virtual activatable elements are notprojected by the wearable extended reality appliance onto thetouch-sensitive surface. For example, the second time period may includea few milliseconds, seconds, minutes, or any other period of time.

Some disclosed embodiments may include deactivating at least onefunctionality of at least a portion of the touch-sensitive surfaceduring a second time period when a different plurality of virtualactivatable elements are projected by the wearable extended realityappliance onto the touch-sensitive surface. For example, the projectionof the different plurality of virtual activatable elements during thesecond time period may include a projection of a new virtual activatableelement at a particular portion of the touch-sensitive surface thatcorresponded to a specific virtual activatable element during the firsttime period, and deactivating the at least one functionality of theparticular portion of the touch-sensitive surface may be a functionalitycorresponding to the specific virtual activatable element. In anotherexample, the projection of the different plurality of virtualactivatable elements during the second time period may include noprojection at a particular portion of the touch-sensitive surface thatcorresponded to a specific virtual activatable element during the firsttime period, and deactivating the at least one functionality of theparticular portion of the touch-sensitive surface may be a functionalitycorresponding to the specific virtual activatable element. Somedisclosed embodiments may include maintaining the at least onefunctionality of the at least a portion of the touch-sensitive surfaceduring a third time period after the first time period and before thedifferent plurality of virtual activatable elements are projected ontothe touch-sensitive surface, in the third time period thetouch-sensitive surface may be outside a field-of-view of the wearableextended reality appliance and therefore the plurality of virtualactivatable elements may not be projected onto the touch-sensitivesurface. In one example, if the touch-sensitive surface reenters thefield-of-view of the wearable extended reality appliance after the thirdtime period and before the different plurality of virtual activatableelements are projected onto the touch-sensitive surface, the pluralityof virtual activatable elements may be projected again onto thetouch-sensitive surface. These embodiments may enable a blind typing(also known as touch typing) like experience using the virtualactivatable elements.

FIG. 31 illustrates a flow chart of an exemplary method that may beexecuted by a processor to perform operations for implementing hybridvirtual keys in an extended reality environment. Method 3100 may includea step 3110 of receiving first signals corresponding to positions on atouch-sensitive surface of a plurality of virtual activatable elementsvirtually projected by a wearable extended reality appliance. Method3100 may also include a step 3111 of determining position of virtualactivatable elements based on the first signals. Further, method 3100may include a step 3112 of receiving touch input of a user via thetouch-sensitive surface. Method 3100 may include a step 3113 ofdetermining location associated with the touch input based on secondsignals generated as a result of the interactions with at least onesensor. In one example, step 3112 and/or step 3113 may occur before,after, or simultaneously with step 3110 and/or step 3111. Method 3100may include a step 3114 of comparing location of touch input withdetermined position. Method 3100 may also include a step 3115 of causinga change in virtual content. Further, in some examples method 3100 mayinclude an optional step 3116 of deactivating functionality oftouch-sensitive surface.

Some disclosed embodiments may include receiving additional signalscorresponding to positions on a keyboard adjacent the touch-sensitivesurface of additional virtual activatable elements virtually projectedby the wearable extended reality appliance on keys of the keyboard. Akeyboard may be located near the touch-sensitive surface. The keyboardmay be located to the left or the right of the touch-sensitive surface.As another example, the keyboard may be located above or below thetouch-sensitive surface. As another example, the keyboard may betouching or sharing a border with the touch-sensitive surface. Thekeyboard may contain one or more physical keys. The keyboard may alsoinclude additional virtual activatable elements in addition to thevirtual activatable elements provided on the touch-sensitive surface. Awearable extended reality appliance may generate signals indicatingpositions associated with the additional virtual activatable elementsprojected onto the keyboard. For example, a keyboard may include blankkeys, and additional virtual activatable elements may be projected ontothe blank keys. As another example, a keyboard may include a sectionwith no keys. Additional virtual activatable elements may be projectedonto the section with no keys.

Some disclosed embodiments may include determining from the additionalsignals the positions of the additional virtual activatable elements onthe keys of the keyboard. A signal may be received that corresponds tothe position of the additional virtually activatable elements on thekeyboard. The processor may determine the position of the additionalvirtual activatable element based on the signal. The processor maydetermine the position of the additional virtual activatable elements onthe keyboard in the same manner in which the processor determines theposition of the plurality of virtual activatable elements on thetouch-sensitive surface. For example, a processor may determine thevirtual activatable element is located on the left side of the keyboard.As another example, the processor may determine the virtual activatableelement is located on the right side of the keyboard.

By way of example, FIG. 32 illustrates an example of a keyboard withadditional virtual activatable elements virtually projected onto thekeys of the keyboard. For example, as illustrated in FIG. 32, keyboard3211 may contain keys 3210. Virtual activatable elements may beprojected onto keys 3210 of keyboard 3211.

Some disclosed embodiments may include receiving a key press input viaat least one key of the keyboard. A key press input may occur whenpressure is applied to a key of the keyboard. A user may apply a forceto one or more keys using a user's hand or part of a user's hand. Forexample, a user may press down on a key using the user's finger, palm,or wrist. The pressure applied by the user to the key may generate asignal indicating that a key of the keyboard has been pressed.

Some disclosed embodiments may include identifying one of the additionalvirtual activatable elements corresponding to the key press input. Asdiscussed above, a user may press a physical key on the keyboard. Insome embodiments, a user may additionally or alternatively press a keyand create a key press input that corresponds to a virtual activatableelement. For example, one or more keys may be projected onto a keyboardand the user may perform a pressing gesture on one of the projectedkeys. As another example, additional virtual activatable elements may beprojected on a set of physical keys and a user may perform a pressinggesture on one of the physical keys. The processor may identify whichvirtual activatable element the user intended to perform based on thekey press input. For example, the user may push the “k” letter key,where the “k” letter key may be an additional virtual activatableelement, creating a key press input. As another example, the user maypush any numbered key, where the numbered key may be an additionalvirtual activatable element, creating a key press input.

Some disclosed embodiments may include causing a second change in thevirtual content associated with the wearable extended reality appliance,wherein the second change corresponds to the identified one of theadditional virtual activatable elements. The processor may cause achange (e.g., second change) in virtual content by adjusting the itemsdisplayed to a user based on, for example, the one or more key pressinputs discussed above. For example, a user may press a virtualactivatable element representing a caps lock key. In response, theprocessor may cause a change in virtual content by capitalizing text ina word document.

Some disclosed embodiments may include receiving a keyboardconfiguration selection and causing the wearable extended realityappliance to virtually project the additional virtual activatableelements to correspond to the selected keyboard configuration. In someembodiments, one or more keys of a physical keyboard may be blank (e.g.,not be marked with any markings like letters or symbols), and those keysmay not correspond to a function. Therefore, when a user presses any ofthose keys, there may be no corresponding change in the virtual contentbeing displayed on the extended reality display. In contrast, when oneor more virtual activatable elements are projected on the blank keys,the physical keyboard may reflect the keyboard configuration associatedwith the projected virtual activatable elements. Thus, for example, ifthe letter “K” is projected onto one of the blank keys, pressing of thatkey may cause the letter K to be displayed on the extended realitydisplay. As another example, the keys of the physical keyboard may havea default function. The keys may contain physical graphics thatcorrespond to the function. For example, a key may contain letters,numbers, or icons. Thus, for example, a keyboard may have a QWERTY orDVORAK layout and pressing one or more keys on the keyboard may cause acorresponding alphanumeric character to be represented on the extendedreality display. When there is no projection onto the keyboard, the keysmay maintain their default functionality. In contrast, when there is aprojection by the wearable extended reality appliance, the physicalkeyboard may reflect the keyboard configuration as projected by thewearable extended reality appliance, and the default functionality maybe turned off. For example, the default configuration of a certain keymay be the letter “K”. The projection of the wearable extended realityappliance may project the letter “I” onto that key of the keyboard. Whenthe projection is active, pressing that key may cause the letter “I” toappear on the extended reality display. In contrast, when the projectionis not active, pressing that same key may cause the default letter “K”to appear on the extended reality display.

Some disclosed embodiments may include selecting the additional virtualactivatable elements based on at least one of a user action, a physicaluser location, a physical location of the wearable extended realityappliance, a physical location of the keyboard, or an event in a userenvironment. Different virtual activatable elements may be availablebased on a user action. For example, a user may open a video file, andbased on the action, virtual activatable elements related to videoediting may be available to the user. As another example, differentvirtual activatable elements may be available based on the physicallocation of the user, the wearable extended reality appliance, and/orthe keyboard. For example, when a user is in a workplace environment,virtual activatable elements related to editing word documents may beavailable to a user. As another example, when a wearable extendedreality appliance is located in a user's home office, virtualactivatable elements related to changing display settings may beavailable to a user. As another example, when a keyboard is located in apublic setting, virtual activatable elements related to volumeadjustment may be available to a user. As another example, differentvirtual activatable elements may be available based on an event in auser environment. For example, a person other than the user may enterthe user's environment. When a person other than a user enters theuser's environment, virtual activatable elements related to sharingdocuments may not be available to the user.

Some disclosed embodiments may include determining whether the user is awearer of the wearable extended reality appliance. The processor maydetermine that an individual wearing the wearable extended realityappliance is the same individual that may be performing operations in anextended reality environment. For example, a keyboard may include acamera. The processor may determine a user who provided input is thewearer of the wearable extended reality appliance based on image datafrom the camera. As another example, the wearable extended realityappliance may include a camera. The processor may determine a user whoprovided input is the wearer of the wearable extended reality appliancebased on image data from the camera.

Some disclosed embodiments may include in response to a determinationthat the user is the wearer of the wearable extended reality appliance,causing the change in the virtual content associated with the wearableextended reality appliance. When the processor determines that the userproviding the touch input is wearing the wearable extended realityappliance, the processor may cause a change in virtual content byadjusting the items displayed to the user based on the touch input. Forexample, a user who is wearing the wearable extended reality appliancemay create a touch input by adjusting a display brightness scroll bar.The processor may determine that the user is wearing smart glasses basedon image data from a camera and by comparing an image of wearer of thewearable extended reality appliance to an image of the user creating theinput. The processor may change the virtual content based on the touchinput by adjusting the brightness because the user is a wearer of thewearable extended reality appliance.

Some disclosed embodiments may include in response to a determinationthat the user is not the wearer of the wearable extended realityappliance, forgoing causing the change in the virtual content associatedwith the wearable extended reality appliance. A user who may not bewearing the wearable extended reality appliance may engage with thetouch-sensitive surface. When a user not wearing the wearable extendedreality appliance provides an input via the touch-sensitive surface, theprocessor may take no action and my not alter the displayed virtualcontent. The processor may determine that the user is not the wearer ofthe wearable extended reality appliance in the same manner the processormay determine that the user is the wearer of the wearable extendedreality appliance. For example, a user may create a touch input byadjusting a volume scroll bar. The processor may determine that the useris not wearing smart glasses based on image data from a camera and bycomparing an image of the wearer of a wearable extended realityappliance to an image of the user creating the input. The processor maynot change the virtual content based on the touch input by not adjustingthe volume because the user is not a wearer of the wearable extendedreality appliance.

In some disclosed embodiments the user may be an individual other than awearer of the wearable extended reality appliance. A touch input may bereceived from someone other than the wearer of the wearable extendedreality appliance. The touch input may be from a user, not wearing thewearable extended reality appliance, engaging with the touch-sensitivesurface. The processor may cause a change in the virtual content basedon the determination that the user is not wearing the wearable extendedreality appliance. For example, a user may be an individual who haspermission to edit a shared word document, but the user may not bewearing a wearable extended reality appliance. The user may create atouch input that corresponds to typing in a shared word document. Theprocessor may change the virtual content by adding the typed word intothe shared word document based on the determination that the user is notwearing the wearable extended reality appliance, but the user does haveediting permission.

In some disclosed embodiments the user is a wearer of a second wearableextended reality appliance and the second wearable extended realityappliance projects a second plurality of virtual activatable elements onthe touch-sensitive surface. The second wearable extended realityappliance may virtually project an overlay on the touch-sensitivesurface that may be displayed to the user using the second wearableextended reality device. The second wearable extended reality appliancemay project virtually activatable elements in the same manner as thefirst wearable extended reality appliance.

Some disclosed embodiments may involve, based on the coordinate locationof the touch input, determining that the touch input corresponds to aparticular virtual activatable element of the second plurality ofvirtual activatable elements. The processor may determine that the touchinput is located in a location that corresponds to a particular virtualactivatable element. The processor may determine the locations of thesecond plurality of virtual activatable elements in the same manner asthe processor determines the location of the plurality of virtualactivatable elements. The particular virtual activatable element may bechosen from the second plurality of virtual activatable elements.

Some disclosed embodiments may include causing a second change in thevirtual content associated with the wearable extended reality appliance,wherein the second change corresponds to the particular virtualactivatable element of the second plurality of virtual activatableelements. A processor may cause a second change in virtual content byadjusting the items displayed to the user. The second change in contentmay be in response to a user, who may be wearing the wearable extendedreality appliance, engaging with the touch-sensitive surface. Forexample, the processor may adjust the size of the display screens. Asanother example, the processor may add additional display screens orremove display screens. As another example, a processor may play audioor display pictures to a user. In some embodiments, the second changemay be based on a virtual activatable element triggered by the user. Forexample, a user may trigger a virtual activatable element thatcorresponds to a zoom in feature. In response, the processor may changethe virtual content by zooming in on the content. As another example, auser may trigger a virtual activatable element that corresponds to adelete feature. In response, the processor may change the virtualcontent by removing an item displayed to the user. In some embodiments,the second change in virtual content may be the same as the first changein virtual content. In another embodiment, the second change in virtualcontent may be different from the first change in virtual content. Inanother embodiment, the second change in virtual content may beassociated with the second wearable extended reality appliance.

Hand gestures are important way of interaction with and control ofextended reality systems and environments. Information about handgestures obtained from images captured using image sensors included inwearable extended reality appliances may be insufficient. For example,when a hand is outside the field of view of the image sensors of thewearable extended reality appliance, or when the hand (or parts of thehand, such as a digit or a tip of a digit) is occluded in the field ofview (for example, by another object or by other parts of the hand),identification of actions performed by the hand or of gestures may bechallenging. Therefore, augmenting the images with data from othersensors, such as sensors of different types or sensors positionedelsewhere, is desired.

In some embodiments, a systems, methods, and a non-transitory computerreadable medium configured for use with a keyboard and a wearableextended reality appliance combination in order to control a virtualdisplay (or any other kind of virtual content, for example of a virtualcontent in an extended reality environment) is disclosed. Thenon-transitory computer readable medium may contain instructions thatmay be executed by at least one processor in order to performoperations. As discussed above, one or more input devices may beconfigured to allow one or more users to input information. For example,in some embodiments, a keyboard or keyboards may be used as the one ormore user input devices that may be configured to allow one or moreusers to input information. As another example, in some embodiments, anintegrated computational interface device in the form of a keyboard maybe configured to allow one or more users to input information. Akeyboard may allow a user to input text and/or alphanumeric charactersusing, for example, one or more keys associated with the keyboard.

By way of example, FIG. 33 illustrates an example of a keyboard and awearable extended reality appliance in combination to control a virtualdisplay, consistent with embodiments of the present disclosure. In someembodiments, virtual display 3310 may include a virtual display screenpresented to a user by wearable extended reality appliance 3312. In someembodiments, keyboard 3311 may be a physical keyboard. In otherexamples, keyboard 3311 may be a virtual keyboard presented to a user bywearable extended reality appliance 3312. Keyboard 3311 may be separatefrom virtual display 3310. In some embodiments, wearable extendedreality appliance 3312 may include a pair of smart glasses, a headmounted display, or any other implementation of a wearable extendedreality appliance discussed herein.

Some embodiments may involve receiving, from a first hand-positionsensor associated with the wearable extended reality appliance, firstsignals representing first hand-movements. A hand-position sensor mayinclude any form of detector configured to determine a placement,location, pose, arrangement, orientation, movement, or any otherphysical characteristic of a human hand. The sensor may output locationinformation, such as coordinates or other location-associatedmeasurements or data. For example, in some embodiments, a sensor maydetect a specific location, position, or displacement from a position.In some embodiments, the sensor may detect a location of a hand in aphysical space. For example, the sensor may provide coordinate locationsrelative to a predetermined reference frame. As another example, thesensor may provide angular positions of the hand relative to apredetermined reference frame. A hand-position sensor may be located onthe wearable extended reality appliance. For example, the hand-positionsensor may be located on the wearable extended reality appliance. Forexample, the sensor may be located on the rim of a pair smart glasses.In some embodiments, the sensor may be located on the temple of the pairof smart glasses. In some embodiments, the sensor may be located on thelenses of the pair of smart glasses. As another example, a hand-positionsensor may be located on the keyboard. In some embodiments, the sensormay be located on the one or more keys associated with the keyboard. Insome embodiments, the sensor may be located in a housing of thekeyboard. In some embodiments, the sensor may be located on a side ofthe keyboard.

In some embodiments, signals (e.g. the first signals) may representhand-movements. Hand-movements may refer to movement of one or moreportions of a user's hand, for example, movement of one or more fingers.By way of example, hand-movements may include changes in a positionand/or pose of a user's fingers as the user types or enters informationusing a keyboard. In some embodiments, the changes in position may occurfrom the user's fingers interacting with the keys of a keyboard. In someembodiments, the changes in position and/or pose may occur from theuser's fingers interacting with the trackpad of the keyboard. In someembodiments, the changes in position and/or pose may corresponds to theuser's fingers interacting with a virtual object, such as a virtualcontrol element, a virtual object located on a surface (such as thesurface that the keyboard is placed on), a virtual object located inmid-air, and so forth. In some embodiments, the changes in positionand/or pose may correspond to a gesturing of the user. As anotherexample, hand-movements may include movement of a user's wrist, such asan up-down or side-to-side motion of a user's hand or a rolling movementof a user's wrist. It is also contemplated that hand-movements mayinclude movement of a portion of the user's fore-arm adjacent to theuser's wrist. It is further contemplated that hand-movements may alsoinclude one or more gestures made by a user with one or more fingers.For example, gestures may include placing one or more fingers on atrackpad (or any other surface) and sliding the one or more fingershorizontally or vertically to scroll, placing two or more fingers on thetrackpad and pinching to zoom in or out, tapping one or more fingers onthe surface, using one or more fingers to press down on the keys of thekeyboard, or tapping one or more fingers on the surface while using oneor more fingers to push down on the keys.

In some embodiments, first signals representing hand-movements may bereceived from a first sensor associated with the wearable extendedreality appliance. For example, first signals may be received from ahand-position sensor associated with smart glasses. In such examples,the hand-position sensor may include, for example, an image sensor, acamera, a depth sensor, a radar, a Lidar, a sonar, or another type oflocation sensor. In some embodiments, the hand-position sensor mayinclude a camera. The camera may capture a series of images of theuser's hand-movements. The hand-movements may include the user's handpositioned at different locations or in different combinations. Theuser's hand may include one or more fingers and/or the user's wrist. Thehand position-sensor may detect a change in location or position of, forexample, the one or more fingers or the wrist based on the cameraimages. The camera may generate a signal representative of the change inlocation or position of one or more portions of the user's hand overtime. In other examples, the camera may generate a signal in combinationwith a processor. In some embodiments, the hand-position sensor mayinclude an image sensor. The image sensor may be configured to captureone or more images. The one or more images captured by the camera or theimage sensor may be analyzed using a gesture recognition algorithm, apose estimation algorithm, or a trained machine learning model todetermine hand related data. The hand related data may includehand-movements, hand location, or hand poses. The first signals receivedfrom the hand-position sensor may represent first hand-movements, mayrepresent hand location data, may represent hand identity data, or mayrepresent hand pose data.

By way of example, FIG. 34 illustrates a first hand-position sensorassociated with a wearable extended reality appliance. For example, asillustrated in FIG. 34, wearable extended reality appliance 3410 mayinclude smart glasses. Wearable extended reality appliance 3410 maycontain first hand-position sensor 3411. First hand-position sensor 3411may be located anywhere on wearable extended reality appliance 3410. Insome embodiments, the hand-position sensor may be located on the temple.In some embodiments, the hand-position sensor may be located on thelens. In some embodiments, the hand-position sensor may be located onthe bridge. In some embodiments, the hand-position sensor may be locatedon the rim. Signals may be received from first hand-position sensor3411. The signals may represent first hand-movements from hand 3412 of auser. In another example, the signals may represent a pose of hand 3412of a user. In yet another example, the signals may represent a positionof at least part of hand 3412 of a user.

In some embodiments, the first hand-movements may include an interactionwith a no-feedback object. In some embodiments, one or more of thehand-movements (e.g., first hand-movements, second hand-movements) mayinclude one or more interactions with a no-feedback object. A feedbackobject may include an object that provides a response to a user. Thetype of response provided by the feedback object may include audiofeedback, tactile feedback, visual feedback, or other responses. In someembodiments, the feedback may be provided through one or more of audiooutput devices, textual output devices, visual output devices, or otherdevices. For example, audio feedback may be provided through one or moreaudio speakers, headsets, and/or other devices capable of producing asound. Tactile feedback, for example, may be provided throughvibrations, motions, shaking, and/or other physical sensations. Visualfeedback may be provided in the form of one or more displays via adisplay screen, an LED indicator, an augmented reality display system,and/or other devices capable of generating a visual display of textualor graphical content. In contrast to a feedback object, a no-feedbackobject may not provide a response to a user when the user is using theno-feedback object. A no-feedback object may include, for example, awriting instrument (e.g. pen) or a fingertip that may write on a virtualscreen projected by a wearable extended reality appliance. Otherexamples of no-feedback objects include non-electronic pointers,furniture, and inanimate, non-electronic objects. Portions of feedbackobjects may constitute no-feedback objects. For example, feedbackobjects have parts that do not provide feedback, and those parts may beconsidered no-feedback. A user may interact with a no-feedback object bytouching the object with the user's hand. For example, a user may toucha wearable extended reality appliance with the user's fingers, and thewearable extended reality appliance may not provide a response to theuser. In some embodiments, a no-feedback object may be or include anobject (or any part of object) not configured to produce an electricalsignal in response to an interaction. In some embodiments, a no-feedbackobject may be or include a nonreactive object (or any nonreactive partor nonreactive surface of an object).

Hand-movements may include actions including interactions with afeedback component. It is contemplated, however, that hand-movements mayinclude actions other than interactions with a feedback component.Actions that include interactions with a feedback component may includeone or more fingers of a user's hand pushing down on one or more keys ofthe keyboard. Interactions with a feedback component may also include,for example, a user's hand scrolling or pinching the touchpad of thekeyboard. Actions other than interactions with a feedback component mayinclude, for example, a user not engaging with the keyboard. Actionsother than interactions with a feedback component may also include, forexample, a user not touching a physical surface. For example, an actionother than an interaction with a feedback component may include a userinteracting with a virtual content (such as a virtual display)positioned in mid-air in an extended reality environment. As anotherexample, an action other than an interaction with a feedback componentmay include a hand-movement outside a field of view of the wearableextended reality appliance. As another example, interactions with afeedback component may include a user's hand clicking or draggingfingers on a physical surface, such as a top surface of a desk or atable.

Some embodiments may involve receiving, from a second hand-positionsensor second signals representing second hand-movements, wherein thesecond hand-movements include actions other than interactions with afeedback component. The second hand-position sensor may have similarstructural and physical characteristics as the first hand-positionsensor, and therefore the details previously described are not fullyrepeated. The second hand-position sensor may also be an image sensor orproximity sensor as described above. The second hand-position sensor maybe located on the keyboard. The sensor may be located on one or morekeys associated with the keyboard (e.g. the space bar and/or thefunction keys) and/or on the trackpad. The sensor may be located oneither side of the keyboard and/or on the top and/or bottom of thekeyboard.

It is also contemplated that the one or more signals generated by thefirst or second hand-position sensor may represent hand location data,hand identity data, or hand pose data. In other examples, the one ormore signals generated by the first or second hand-position sensor mayrepresent hand movement data. Hand location data may represent arelative location of a user's hand. For example, hand location data mayrepresent a displacement of a user's hand measured relative to aspecific point in space. For example, the digital signal associated withhand movement may represent a distance between a user's hand relative tothe keyboard. Additionally, or alternatively, hand location data mayrepresent an absolute location of a user's hand. For example, handlocation data may be designated using a specific set of coordinatepositions. Hand identity data may include information associated with aportion of a user's hand. For example, hand identity data may includeposition, velocity, acceleration, or other information indicating that auser's finger is performing an action. In another example, hand identitydata may include position, velocity, acceleration, or other informationindicating a user's wrist is performing an action. In some examples,hand identity data may be used to determine whether the hand is a handof the user of the wearable extended reality appliance. As anotherexample, hand identity data may be used to determine that the hand isnot a hand of the user of the wearable extended reality appliance. Handpose data may represent gestures made by a user with one or morefingers. For example, one or more digital signals generated by the firstor second hand-position sensor may represent one or more gestures madeby the user. Such gestures may include, for example, scrolling,pinching, tapping, and/or pressing using one or more fingers, and orother combinations of movements involving one or more fingers, wrist,and/or a forearm of the user.

In some embodiments, second signals representing hand-movements may bereceived from a second sensor associated with the keyboard. For example,a physical keyboard may include a sensor (an infrared proximity sensor,etc.) configured to detect movement of one or more portions of a user'shand over the keyboard. In some embodiments, the second hand-positionsensor may include a proximity sensor configured to determine positionof one or more portions of a user's hand when the hand is hovering overthe keyboard. Inputs from such sensors may be used to provide visualindications of one or more gestures near the keyboard, for example, ofgestures indicating a likelihood that a user may press a particular keyor press one or more keys of a particular group of keys. For example, adepiction of a keyboard may be provided visually on a screen. Adepiction of a keyboard may also be shown through a wearable extendedreality appliance. The visual indication may be shown over the depictionof the keyboard. For example, one visual indication may be used toindicate a pressing of a key or of a group of keys, and another visualindication may be used to indicate a likelihood of an intention of auser to press a particular key, or indicating a likelihood of anintention of a user to press a one or more keys of a particular group ofkeys.

By way of example, FIG. 35 illustrates an example of a secondhand-position sensor associated with a keyboard 3510 with second signalsrepresenting second hand-movements, consistent with some embodiments ofthe present disclosure. Keyboard 3510 may include second hand-positionsensor 3511. Signals, representing hand-movements from hand 3512 of auser may be received from second hand-position sensor 3511.

In some embodiments, the keyboard may be located on a surface, and whenlocated on a surface, the second hand-movements may include aninteraction with the surface. In some embodiments, the surface may be atop surface of a desk. In some embodiments, the surface may be a topsurface of a table. In some embodiments, the surface may be a floor. Byway of example, a virtual controller or a widget may be displayed on thesurface, for example by the wearable extended reality appliance. Avirtual controller may include, for example, a volume bar, a brightnessadjuster, or any other control that may allow a user to control acharacteristic of the virtual display (or of any other kind of virtualcontent, for example of virtual content in an extended realityenvironment) or of the wearable extended reality appliance. The secondsignals may be analyzed to determine whether a hand is touching thesurface. In response to the determination that the hand is touching thesurface, an action may be performed. The actions may include controllingone or more characteristics (e.g., brightness, volume) associated withthe virtual display (or with any other kind of virtual content, forexample with a virtual content in an extended reality environment). Insome embodiments, the keyboard may include a particular surface that maybe configured to be substantially perpendicular to a first surface whenthe keyboard is placed on the first surface, and the secondhand-position sensor may be included in the particular surface. In someembodiments, the keyboard may include at least two surfaces that areconfigured to be substantially perpendicular to the first surface, theat least two surfaces may include a surface closer to a space bar and asurface farther from the space bar, and the particular surface may bethe surface farther from the space bar. The second hand-movements mayinclude one or more interactions with either surface. The secondhand-position sensor may include an image sensor and a field of view ofthe image sensor may include at least part of the surface, and thereforethe processor may receive signals from the sensor representinghand-movements on either surface.

In some embodiments, at least one of the hand-position sensor may be animage sensor. The term “image sensor” is recognized by those skilled inthe art and refers to any device configured to capture images, asequence of images, videos, and so forth. Image sensors may includesensors that convert optical input into images, where optical input canbe visible light (like in a camera), radio waves, microwaves, terahertzwaves, ultraviolet light, infrared light, x-rays, gamma rays, and/or anyother type of electromagnetic radiation. Image sensors may include 2Dand 3D sensors. Examples of image sensor technologies may include: CCD,CMOS, or NMOS. 3D sensors may be implemented using differenttechnologies, including: stereo camera, active stereo camera, time offlight camera, structured light camera, radar, range image camera, andother technologies capable of generating a display representing anobject or a scene in three dimensions. The first signal may includeimage data in the form of one or more images captured using one or moreimage sensors. The one or more images may be analyzed using a machinelearning model trained using training examples to determine hand relateddata (such as hand movements data, hand location data, hand pose data,hand-surface touch status data, and other data). Such training examplesmay include sample images together with corresponding hand related dataindicating particular hand locations, hand poses, hand-surface touchstatuses, or other data.

In some embodiments, at least one of the hand-position sensor may be aproximity sensor. The term “proximity sensor” may include any deviceconfigured to detect the presence of an object without physical contactwith the object. Proximity sensors may use sound, light, infraredradiation (IR), or electromagnetic fields to detect a target.

In some embodiments, the one or more position sensors (e.g., handposition sensors) may generate one or more digital signals. The digitalsignals may consist of one, two, or any number of signals (e.g., firstsignals, second signals). Thus, for example, a first hand-positionsensor may be configured to generate a first signal, which may be adigital signal. In some embodiments, the sensors may consist of one,two, or any number of sensors (e.g., first sensor, second sensor).

In some embodiments, a type of the second hand-position sensor differsfrom a type of the first hand-position sensor. As discussed above, theremay be many different types of hand-position sensors. It is contemplatedthat in some embodiments, different types of hand-position sensors maybe associated with different components of some disclosed embodiments.Different types of hand-position sensors may be selected so that thesensors may provide different types of signals. For example, the firsthand-position sensor may be an image sensor, and the secondhand-position sensor may be a proximity sensor. The first hand-positionsensor may include an image sensor and may be configured to generatesignals representative of hand-identity data. The second hand-positionsensor, however, may include a proximity sensor configured to generatesignals representative of hand-location data. By way of another example,the first hand-position sensor may be a proximity sensor and thesecond-hand position may be an image sensor. In some embodiments, thefirst hand-position sensor and the second hand-position sensor may be ofthe same type. For example, the first hand-position sensor and thesecond hand-position sensor may both include image sensors. As anotherexample, the first hand-position sensor and the second hand-positionsensor may both include proximity sensors.

FIG. 36 illustrates an example configuration in which a type of secondhand-position sensor differs from a type of first hand-position sensor.For example, as illustrated in FIG. 36, wearable extended realityappliance 3613 may include a pair of smart glasses. Wearable extendedreality appliance 3613 may contain first hand-position sensor 3614 thatmay be a camera. As also illustrated in FIG. 36, keyboard 3610 maycontain second hand-position sensor 3611 that may be a proximity sensor,which may be different from first hand-position sensor 3614. Secondhand-position sensor 3611 may generate signals representing secondhand-movements from hand 3612 of a user.

In some embodiments, the keyboard may include an associated input area,including a touch pad and keys, and wherein the operations may furtherinclude detecting the second hand-movements in the area outside theinput area (for example, in an area including no keys, touchpads,trackpads, joysticks, or other forms of touch sensors). The keyboard maybe configured to receive input from an input area and/or from the firsthand-position sensor and the second hand-position sensor. The keyboardmay include many different areas. In some embodiments, the keyboard mayinclude one or more keys in the input area, for example including astandard keyboard such as QWERTY, Dvorak, or any other type of keyboardlayout. The keyboard may include additional keys in the input area. Forexample, the one or more additional keys may include numeral entry keysor keys with math symbols. In some embodiments, the input area mayfurther include a fingerprint reader. In some embodiments, the inputarea may include a trackpad and/or touchpad. A user may performhand-movements that are outside of the input area associated with thekeyboard. For example, the user may perform hand-movements that are notassociated with the input area (for example, in an area including nokeys, touchpads, trackpads, joysticks, or other forms of touch sensors).As one example, the user may touch a nonreactive surface of the keyboardor a nonreactive surface in the environment of the keyboard (such as anonreactive surface that the keyboard is placed on). In another example,the user's hand-movements may occur in an area that is not near anyportion of the keyboard.

By way of example, FIG. 37 illustrates an example of a keyboard thatincludes an associated input area including a touch pad and keys. Forexample, as illustrated in FIG. 37, keyboard 3710 may include input areaincluding keys and touchpad 3711. Keyboard 3710 may be configured toreceive input from one or more keys and/or touchpad 3711. In someembodiments, keyboard 3710 may include second hand-position sensor 3712.Second hand-position sensor may generate signals representing secondhand-movements of hand 3713 of a user. As illustrated in FIG. 37, hand3713 of a user may produce hand-movements that are in an area outside ofthe input area (not involving the keys and/or touchpad 3711). Forexample, hand 3713 may be making a gesture without touching any portionof keyboard 3710.

Some embodiments may involve controlling the virtual display (or anyother kind of virtual content, for example virtual content in anextended reality environment) based on first and second signals. As usedin this disclosure, controlling the virtual display may include alteringan appearance or content of the virtual display. For example, the atleast one processor may control the virtual display by adjusting one ormore display settings. Adjusting one or more display settings mayinclude, for example, adjusting the brightness, color temperature,sound, size of windows, fonts, resolution, pixel size, location,orientation, and so forth. Altering an appearance or content of thevirtual display may include, for example, deleting or adding newwindows, moving windows on or off the screen, or adding or deletingelements from the display. In some embodiments, the virtual display maybe controlled based on the first and/or second signals. The processormay receive the first and/or second signals from one or more sensorsassociated, for example, with a keyboard or with a wearable extendedreality appliance. Based on the received signals, the processor maycontrol the virtual display (or any other kind of virtual content, forexample controlling virtual content in an extended reality environment).For example, a processor may determine that a signal representing ahand-movement includes a user attempting to zoom in on the virtualdisplay. The processor may determine that the hand-movement represents azooming motion based on the user's hand gestures. The processor maycontrol the virtual display by zooming in on a window based on thesignals representing the hand-movement.

Some embodiments may involve controlling the virtual display based onthe first signals and the second signals when a certainty levelassociated with at least of one of the first hand-movements or thesecond hand-movements is above a threshold. A certainty level may referto a probability or degree of confidence in the determination that auser's hand moved in some manner. In some embodiments, the certaintylevel may be based on a magnitude of a distance or an angle by whichsome or all portions of the user's hand move from an initial position.For example, a sensor may be configured to detect a hand-movement onlywhen a magnitude of the hand-movement (e.g., distance or angle by whichthe hand moved) is greater than a threshold magnitude. For example, thesensor may detect a hand movement, when a position of a user's handchanges by 1 mm, 5 mm, 10 mm, or any other desired unit of length. Asanother example, the sensor may detect a hand movement, when a user'shand is rotated by an angle of 1°, 5°, 10°, or any other desired angle.When the sensor determines that the hand-movements are below thethreshold magnitude, the sensor may not generate a signal. When thehand-movement exceeds the threshold magnitude (e.g., 1 mm, 5 mm, 10 mm,or 1°, 5°, 10°), the sensor may generate and send a signal representinga hand-movement to the processor.

Some embodiments may involve analyzing the second signals to identify afirst portion of the second signals associated with low ambientmovements and a second portion of the second signals associated withhigh ambient movements. In one example, ambient movements may be orinclude movements of objects other than a hand (or other than the handof the user). In some examples, a machine learning model may be trainedusing training examples to identify low and high ambient movements ofsignals. An example of such training example may include a sampleportion of a signal, together with a label indicating whether the sampleportion corresponds to low ambient movements or to high ambientmovements. The trained machine learning model may be used to analyze thesecond signals and identify the first portion of the second signalsassociated with low ambient movements and the second portion of thesecond signals associated with high ambient movements. In some examples,an entropy of a portion of the second signals may be calculated, and thecalculated entropy of the portion of the second signals may be comparedwith a selected threshold to determine whether the portion of the secondsignals corresponds to low ambient movements or to high ambientmovements. In one example, the selected threshold may be selected basedon a duration of time corresponding to the portion of the secondsignals. In another example, the selected threshold may be selectedbased on a distance of the hand from the second hand-position sensorcorresponding to the portion of the second signals. In some examples,the virtual display (or any other kind of virtual content, for examplevirtual content in an extended reality environment) may be controlledbased on the first signals, the second signals, and the identificationof the first portion and the second portion of the second signals. Forexample, the virtual display (or any other kind of virtual content, forexample virtual content in an extended reality environment) may becontrolled based on the first signals and the first portion of thesecond signals, regardless of the second portion of the second signals.In another example, a different weight may be assigned to the first andsecond portions of the second signals, and the control of the virtualdisplay may be based on the first signals and on a weighted function ofthe two portions of the second signals. In some examples, the weightsmay be selected based on an amount of ambient movement associated witheach portion.

Some embodiments may further involve determining three-dimensionalpositions of at least part of a hand based on the first signals and thesecond signals and controlling the virtual display (or of any other kindof virtual content, for example controlling virtual content in anextended reality environment) based on the determined three-dimensionalpositions of the at least part of the hand. Three-dimensional positionsmay refer to coordinate locations relative to a set of coordinate axes.For example, a processor may determine a position and/or orientation ofa part of a user's hand using three coordinate values (i.e. x, y, z)with reference to a set of Cartesian coordinate axes. As anotherexample, a processor may determine the position and/or orientation of apart of the user's hand using polar coordinate values including a radiusand two different planes perpendicular to each other. In someembodiments, the three-dimensional position may include a position of auser's finger, wrist, or any other part of the hand. The processor mayreceive signals based on the hand-movements of a user to determine whereat least part of the user's hand is located relative to a referencelocation or relative to a predetermined set of coordinate axes. Theprocessor may determine which part of the hand is performing a specificmotion. For example, the processor may determine whether the fingers,the wrist, or any other portion of the user's hand is performing aspecific movement. The processor may control the virtual display (or anyother kind of virtual content, for example controlling virtual contentin an extended reality environment) based on the specific motion. Forexample, a processor may determine that a thumb and forefinger arepinching together to perform a zoom in motion. The processor may zoom inon the virtual display based on the zoom-in motion.

In some embodiments, controlling the virtual display based on the firstand second signals may involve controlling a first portion of thevirtual display based on the first signals and controlling a secondportion of the virtual display based on the second signals. A virtualdisplay may include, for example, one or more objects, textual displays,graphical displays, windows, or icons. The processor may be configuredto control one or more of these items (e.g., first portion of virtualdisplay) based on signals received from the first sensor. The processormay also be configured to control other of the items (e.g., secondportion of virtual display) based on signals received from the secondsensor. In some embodiments, the first signals and second signals mayrepresent different hand-movements or characteristics associated withthe user's hand. For example, the at least one processor may determinethat a hand is the hand of the user based on the first signals. The atleast one processor may also determine that the user's hand is touchinga surface based on the second signals. As another example, the at leastone processor may determine that the user's hand is performing a certaingesture based on the second signals. Based on these determinations, forexample, the at least one processor may control a first portion of avirtual display based on the first signals and a second portion of thevirtual display based on the second signals. For example, the at leastone processor may determine that a hand is the hand of the user and maygive the hand permission to interact with the virtual display. Forexample, the at least one processor may select a window as a firstportion of the virtual display based on the first signals. The processormay select a window as the first portion based on the user's handtouching a surface that represents a window. The processor may adjustthe size of the window as the second portion of the virtual displaybased on the second signals. The processor may adjust the size of thewindow as the second portion based on the user's hand performing agesture that represents a zoom in or out motion. As another example, thehand movement detected by the wearable extended reality appliance maycontrol an input for a widget. The hand movement detected by thekeyboard may control an input for another widget. In some embodiments,the first signals and second signals may represent the samehand-movements. For example, the at least one processor may determinethat a user's hand is touching a surface based on the first signals. Theat least one processor may further determine that the user's hand istouching the same surface based on the second signals. Based on thesedeterminations, for example, the at least one processor may control afirst and second portion of the virtual display by selecting a window,based on the first and second signals.

In some embodiments, the first portion and the second portion partiallyoverlap. Thus for example, at least some part of each of the twoportions of the virtual display that may be controlled based on thefirst and second signals may occupy the same positions on the virtualdisplay. For example, the first and second signals may be analyzed todetermine that a user's hand is touching a surface. A processor maydetermine that the first and second portion of the virtual displayrelate to a window in the virtual display. The first portion and thesecond portion may include the same window in the virtual display. Asanother example, a processor may determine that the first and secondportion of the virtual display relate to two windows. The two windowsmay include portions that partially overlap. For example, a portion ofone window may be overtop of a portion of another window.

In some embodiments, the first portion and the second portion arenon-overlapping. For example, the first signals may be analyzed todetermine that a user's hand is touching a surface. The second signalsmay be analyzed to determine that a user's hand is touching a key of thekeyboard. The processor may control a first portion of the virtualdisplay based on the first signals. For example, the processor mayselect a window as the first portion of the virtual display. One or moreportions of the virtual display outside the selected window may form thesecond portion of the virtual display. The processor may control asecond portion of the virtual display based on the second signals. Forexample, the processor may display text in the second portion of thevirtual display outside the window selected based on the first signalsin response to one or more keys selected by the user's hand.

In some embodiments, when at least one of the first hand-movements andthe second hand-movements are detected by the second hand-positionsensor and undetected by the first hand-position sensor, the operationsmay further include controlling the virtual display (or any other kindof virtual content, for example controlling virtual content in anextended reality environment) based only on the second signals. Forexample, at least one of the first hand-movements or the secondhand-movements may be detected by the second hand-position sensor butmay remain undetected by the first hand-position sensor. As one example,this may occur when the second hand-movements are outside a field ofview of the first hand-sensor or when a portion of a user's hand isoutside a field of view of the first hand-sensor. Although the secondhand-movements may not be detectable by the first hand-sensor, they maybe detected by the second hand-sensor. In such situations, the at leastone processor may be configured to control the virtual display basedonly on the second signals.

In some embodiments, a keyboard may include a plurality of keys andoperations may involve analyzing the second signals to determine a userintent to press a particular key of the plurality of keys and causingthe wearable extended reality appliance to provide a virtual indicationrepresenting the particular key based on the determined user intent. Forexample, a processor may use the second signals to identify a key that auser may intend to press out of a plurality of keys. The processor maydetermine a user intent to press a particular key based on the proximityof the user's hand to a key. In another example, a processor maydetermine a user intent to press a particular key based on movement of auser's hand around a particular key. Based on the user intent, theprocessor may provide a sign to the user. The sign may indicate what keythe user was planning to press out of the plurality of keys. The signmay be a visual indication. The visual indication may be providedthrough audio, visual, or tactile means. The visual indication may beprovided through a sign, symbol, icon, letter, or graphical image. Inanother example, the visual indication may be provided by playing asound file or through other audio cues.

In some embodiments, the keyboard may include a plurality of keys, andoperations may involve analyzing the second signals to determine a userintent to press at least one of the group of keys out of the pluralityof keys causing the wearable extended reality appliance to provide avirtual indication representing the group of keys based on thedetermined user intent. For example, a processor may be unable todetermine a specific key that a user intends to press based on analysisof the second signals received from the second sensor. In some examples,the processor may be unable to determine a specific key because of theproximity of the user's hand to multiple keys. The processor maydetermine that the user intended to press at least one key out of agroup of keys. The processer may determine the user intent based onproximity of certain keys. In another example, the processer maydetermine user intent based on the likelihood of certain keys being usedtogether. For example, the processor may be able to determine words thatthe user is typing and determine the user intent based on letters thatare likely to be used together to create the determined words. Asanother example, the group of keys may be keys that have similarfunctions. As another example, the group of keys may be keys that arelocated near each other. In response to the processor being unable todetermine the user intent, the processor may display the group of keysto the user. The display may be through a visual indication. The visualindication may be provided through audio, visual, or tactile means asdiscussed above.

In some embodiments, when the wearable extended reality appliance isunconnected to the keyboard, operations may further involve controllingthe virtual display (or any other kind of virtual content, for examplecontrolling virtual content in an extended reality environment) basedonly on the first signals. For example, the wearable extended realityappliance may be separate from the keyboard. The wearable extendedreality appliance may be physically or electronically separate from thekeyboard. The wearable extended reality appliance may be physicallyseparated from the keyboard when there is no mechanical element, such asa wire or cable or any other mechanical structure, attaching theappliance to the keyboard. The wearable extended reality appliance maybe electronically separated from the keyboard when there is no exchangeof data or communication between the appliance and the keyboard. In someembodiments, the wearable extended reality appliance may be wirelesslyconnected to the keyboard. The wearable extended reality appliance andthe keyboard may be located in different areas. When the wearableextended reality appliance is separated from the keyboard, a user may beable to move freely in a space. The wearable extended reality appliancemay also be used between two or more different users.

Some embodiments may further involve analyzing the second signals todetermine that a hand is touching a part of a physical object associatedwith a virtual widget, analyzing the first signals to determine whetherthe hand belongs to a user of the wearable extended reality appliance,in response to a determination that the hand belongs to the user of thewearable extended reality appliance, executing an action associated withthe virtual widget, and in response to a determination that the handdoes not belong to the user of the wearable extended reality appliance,forgoing performing the action associated with the virtual widget. Inusing the system, a user may interact with a physical object. Thephysical object may be an article located in the physical space aroundthe user. The physical space may include a table, chair, keyboard, orappliance that is within reach of the user. For example, a physicalobject may be a pen, pointer, a keyboard, or any object that a user maybe able to hold. The physical object may be linked to a widget. A widgetmay be a module (physical or virtual) on the interface of a device thatallows the user to perform a function. A module may include a window,icon, image, or other graphic object displayed on the virtual display.The processor may determine that the hand touching the physical objectmay belong to the user of the wearable extended reality appliance. Whenthe hand does belong to the user, the processor may perform an actionbased on the user's interaction with the physical object. The action maycontrol the virtual display. For example, when the user's hand performsa pinching motion, the processor may adjust the size of the widget. Whenthe hand does not belong to the user, the processor may not perform anaction, even if the user interacts with the physical object.

Some embodiments may further involve analyzing the second signals todetermine a location where the hand is touching the physical object andusing the determined location to select the action associated with thevirtual widget. For example, the physical object may contain differentlocations that relate to different actions. A physical object maycontain locations on the top, bottom, front, back, left side or rightside of the object. For example, the top of the physical object mayassociate to a delete function. As another example, the bottom of thephysical object may associate to an add function. The hand-positionsensor may generate signals representing the location of where theuser's hand is touching the physical object. The processor may associatethe location with an action. In one example, the operations may performthe action in response to a first determined location. In anotherexample, the operations may forgo performing the operation in responseto a second determined location.

Some embodiments may further involve determining an orientation of thekeyboard and adjusting display settings associated with the virtualdisplay based on the orientation of the keyboard. An orientation may bea position in physical or virtual space. An orientation may also be aspatial relationship relative to another object in physical or virtualspace. The determination of the orientation may be based on the firstsignals, the second signals, or data from other sensors. In someembodiments, first signals representing first hand-movements maydetermine an orientation of the keyboard. For example, first signals mayrepresent a user's hand typing on a keyboard. The first signals maydetermine an orientation of the keyboard based on the one or more keyspressed by the user's hand. In another embodiment, second signalsrepresenting second hand-movements may determine the orientation of thekeyboard. For example, second signals may represent the user's handscrolling on a trackpad of the keyboard. The second signals maydetermine an orientation of the keyboard based on where a trackpad islocated on the keyboard. For example, the trackpad may be locatedadjacent to a bottom edge of the keyboard. A trackpad may additionallyor alternatively be located adjacent to a top edge of the keyboard. Insome embodiments, a trackpad may additionally or alternatively belocated on a left or right side of the keyboard. In some embodiments,one or more display settings associated with the virtual display may beadjusted based on a determined orientation of the keyboard. For example,the processor may determine, based on the determined orientation, thatthe virtual display should be oriented in a certain way relative to thekeyboard. For example, when the virtual display is not in a certainposition with the keyboard, the processor may move the windows on thedisplay setting to be in a position with the keyboard.

In some embodiments, the wearable extended reality appliance may beselectively connectable to the keyboard via a connector located on aside closest to the space bar. The wearable extended reality appliancemay be attached or detached to the keyboard via a connector. A connectormay be any type of mechanical fastener for attaching the wearableextended reality appliance to the keyboard. In some embodiments, theconnector may include a retractable cable connectable to the wearableextended reality appliance. The connector may also include a rigidcable. The connector may be placed at a side of the keyboard. Forexample, the connector may be located adjacent to a top edge of thekeyboard, adjacent to a bottom edge of the keyboard, on a side farthestfrom the spacebar, or in any other portion of the keyboard. Theconnector may be located on a side closest to the space bar to preventinterference with other items in the user's physical space. When thewearable extended reality appliance may be connectable to a keyboard viaa connector, the wearable extended reality appliance may be secured tothe keyboard. The wearable extended reality appliance may not be at riskof being separated from the keyboard. When the wearable extended realityappliance may be connectable to a keyboard, the appliance may also sendsignals to the keyboard.

By way of example, FIG. 38 illustrates a wearable extended realityappliance selectively connectable to a keyboard via a connector. Forexample, as illustrated in FIG. 38, wearable extended reality appliance3812 may include a pair of smart glasses. Wearable extended realityappliance 3812 may be selectively connectable to keyboard 3810 viaconnector 3811. Connector 3811 may be located on a side closest to spacebar 3813.

Some disclosed embodiments may include a non-transitory computerreadable medium for integrating a moveable input device with a virtualdisplay projected via a wearable extended reality appliance. In someembodiments, when a user moves a moveable input device (e.g., akeyboard) while a wearable extended reality appliance (e.g., smartglasses) is in use, the virtual display projected through the wearableextended reality appliance may change to reflect the change inorientation and/or position of the moveable input device.

Some disclosed embodiments may relate to non-transitory computerreadable medium containing instructions for integrating a moveable inputdevice with a virtual display projected via a wearable extended realityappliance, the computer readable medium containing instructions thatwhen executed by at least one processor cause the at least one processorto perform various steps. Non-transitory computer readable medium mayrefer to any type of physical memory on which information or datareadable by at least one processor can be stored as discussed herein.Examples include Random Access Memory (RAM), Read-Only Memory (ROM),volatile memory, nonvolatile memory, hard drives, CD ROMs, DVDs, flashdrives, disks, any other optical data storage medium, any physicalmedium with patterns of holes, a PROM, an EPROM, a FLASH-EPROM or anyother flash memory, NVRAM, a cache, a register, any other memory chip orcartridge, and networked versions of the same. As used in this context,integrating may include, for example, joining, incorporating, or usingany other method of combining one thing with another so they become awhole.

A moveable input device may include any physical device that may allow auser to provide one or more inputs, and which includes at least aportion that is movable from an initial position to an alternateposition. The disclosed moveable input device may be configured toprovide data to a computational device as discussed herein. The dataprovided to the computational device may be in a digital format and/orin an analog format. Some examples of the moveable input device mayinclude a button, a key, a keyboard, a computer mouse, a touchpad, atouchscreen, a joystick, or another mechanism from which input may bereceived. For example, in some embodiments, a user may provide one ormore inputs via a moveable input device by pressing one or more keys ofa keyboard. As another example, a user may provide one or more inputsvia a moveable input device by changing a position of a joystick bylinear or rotational motion of the joystick. By way of another example,a user may provide one or more inputs via a moveable input device byperforming one or more gestures (e.g., pinch, zoom, swipe, or otherfinger movements) while touching a touchscreen.

A wearable extended reality appliance may include any type of device orsystem that may be worn by or attached to a user for enabling the userto perceive and/or interact with an extended reality environment. Theextended reality appliance may enable the user to perceive and/orinteract with an extended reality environment through one or moresensory modalities. Some non-limiting examples of such sensorymodalities may include visual, auditory, haptic, somatosensory, andolfactory. One example of the extended reality appliance is a virtualreality appliance that enables the user to perceive and/or interact witha virtual reality environment. Another example of the extended realityappliance is an augmented reality appliance that enables the user toperceive and/or interact with an augmented reality environment. Yetanother example of the extended reality appliance is a mixed realityappliance that enables the user to perceive and/or interact with a mixedreality environment.

Consistent with one aspect of the disclosure, the extended realityappliance may be a wearable device, such as a head-mounted device, forexample, smart glasses, smart contact lens, headsets, or any otherdevice worn by a human for purposes of presenting an extended reality tothe human. Typical components of wearable extended reality appliancesmay include at least one of: a stereoscopic head-mounted display, astereoscopic head-mounted sound system, head-motion tracking sensors(such as gyroscopes, accelerometers, magnetometers, image sensors,structured light sensors, etc.), head mounted projectors, eye-trackingsensors, and additionally components described below. Consistent withanother aspect of the disclosure, the extended reality appliance may bea non-wearable extended reality appliance. Specifically, thenon-wearable extended reality appliance may include multi-projectedenvironment appliances.

A virtual display may refer to any type of data representation that maybe displayed by the extended reality appliance to the user as discussedherein. The virtual display may include a virtual object, inanimatevirtual display, animate virtual display configured to change over timeor in response to triggers, virtual two-dimensional content, virtualthree dimensional content, a virtual overlay over a portion of aphysical environment or over a physical object, a virtual addition to aphysical environment or to a physical object, a virtual promotioncontent, a virtual representation of a physical object, a virtualrepresentation of a physical environment, a virtual document, a virtualcharacter or persona, a virtual computer screen, a virtual widget, orany other format for displaying information virtually. Consistent withthe present disclosure, the virtual display may include any visualpresentation rendered by a computer or a processing device. In oneembodiment, the virtual display may include a virtual object that is avisual presentation rendered by a computer in a confined region andconfigured to represent an object of a particular type (such as aninanimate virtual object, an animate virtual object, virtual furniture,a virtual decorative object, virtual widget, or other virtualrepresentation). The rendered visual presentation may change to reflectchanges to a status object or changes in the viewing angle of theobject, for example, in a way that mimics changes in the appearance ofphysical objects. In another embodiment, the virtual display may includea virtual computer screen configured to display information. In someexamples, a virtual display may be a virtual object mimicking and/orextending the functionality of a physical display screen. For example,the virtual display may be presented in an extended reality environment(such as a mixed reality environment, an augmented reality environment,a virtual reality environment, etc.), for example using an extendedreality appliance (such as a wearable extended reality appliance). Inone example, a virtual display may present content produced by a regularoperating system that may be equally be presented on a physical display.In one example, a textual content entered using a keyboard (for example,using a physical keyboard, using a virtual keyboard, etc.) may bepresented on a virtual display in real time as the textual content istyped. In one example, a cursor may be presented on a virtual display,and the cursor may be controlled by a pointing device (such as aphysical pointing device, a virtual pointing device, a computer mouse, ajoystick, a touchpad, a physical touch controller, and so forth). In oneexample, one or more windows of a graphical user interface operatingsystem may be presented on a virtual display. In one example, contentpresented on a virtual display may be interactive, that is, it maychange in reaction to actions of users. In one example, a presentationof a virtual display may include a presentation of a screen frame, ormay include no presentation of a screen frame.

Some disclosed embodiments may involve receiving motion signalsassociated with the moveable input device, the motion signals reflectingphysical movement of the moveable input device. As used in this context,motion signals may include, for example, one or more values thatrepresent a change in position, orientation, angle, direction,arrangement, configuration, speed, acceleration, or any other measure ofrelative position. As used in this context, physical movement may referto a change of position, elevation, orientation, or rotation of themoveable input device. In some examples, motion signals of the moveableinput device (e.g., a keyboard) may be received by a processor which mayreflect physical movement of the moveable input device. For example, theprocessor may receive one or more motion signals that reflect a physicalchange in position or orientation from memory, from an external deviceoutputting data, from values captured using a sensor (e.g., an imagesensor capturing a picture of the physical change), or any other similarprocess of analyzing data. Non-limiting examples of physical movementmay include, for example, a user displacing the keyboard by one inch,one foot, one yard, or by any other distance, a user rotating thekeyboard by 1°, 10°, 90°, 180°, 270°, 360°, or by any other angle, auser pressing any key on the keyboard, a user tilting any one side ofthe keyboard in any direction, or any other change in place, position,orientation, or rotation. For example, a wearable extended realityappliance's (e.g., smart glasses) processor may receive motion signalsthat may include a series of values representing a change in position ororientation from the keyboard reflecting a rotation, displacement,translation, or any other change in a position or orientation of some orall portions of the keyboard.

FIG. 45A illustrates an exemplary set of instructions 4500 to beexecuted by at least one processor for integrating a moveable inputdevice 3902 with a virtual display 4112 projected via a wearableextended reality appliance (e.g., smart glasses). In some embodiments,the operations may be configured to include step 4502 for receivingmotion signals associated with the moveable input device 3902 (e.g., akeyboard), the motion signals reflecting physical movement of themoveable input device. For example, the wearable extended realityappliance's processor may receive motion signals that may include aseries of values representing a change in position or orientation, fromthe moveable input device 3902, for example reflecting rotation of themoveable input device 3902 by a particular angle (such as an anglesmaller than 30°, an angle larger than 30°, 90°, etc.), reflectingdisplacement of the moveable input device 3902, and so forth.

Consistent with some disclosed embodiments, the motion signals of themoveable input device may be determined based on an analysis of datacaptured using at least one sensor associated with the input device. Theat least one sensor may include, for example, a detector, instrument, orother device which measures a physical property of the moveable inputdevice. The physical property measured by the at least one sensor mayinclude a change in position, a velocity, an acceleration, a change inresistance, or a change in any other physical property associated withthe moveable input device. According to some embodiments, an analysis ofdata may refer to a central processing unit (e.g., a processor)executing a sequence of stored instructions (e.g., a program) whichtakes inputs from the moveable input device, process the inputs, andoutput the results to an output device.

Advantageously, in some embodiments, a wearable extended realityappliance may include one or more sensors. The one or more sensors mayinclude one or more image sensors (e.g., configured to capture imagesand/or videos of a user of the appliance or of an environment of theuser), one or more motion sensors (such as an accelerometer, agyroscope, a magnetometer, or any other similar sensor), one or morepositioning sensors (such as GPS, outdoor positioning sensor, indoorpositioning sensor, or any other similar sensor), one or moretemperature sensors (e.g., configured to measure the temperature of atleast part of the appliance and/or of the environment), one or morecontact sensors, one or more proximity sensors (e.g., configured todetect whether the appliance is currently worn), one or more electricalimpedance sensors (e.g., configured to measure electrical impedance ofthe user), one or more eye tracking sensors, such as gaze detectors,optical trackers, electric potential trackers (e.g., electrooculogram(EOG) sensors), video-based eye-trackers, infra-red/near infra-redsensors, passive light sensors, or any other technology capable ofdetermining where a human is looking or gazing.

In some embodiments, the motion signals associated with the moveableinput device may be based on an analysis of the data captured from amotion sensor. For example, a vision or image sensor may capture aplurality of images of the moveable input device. A processor associatedwith the vision or image sensor may be configured to analyze the imagesand determine that a position or orientation of some or all portions ofthe moveable input device has changed.

Consistent with some disclosed embodiments, the motion signalsassociated with the moveable input device may be determined based on ananalysis of images of the moveable input device. As discussed above, animage may refer to a frame, group of pixels, figure, illustration,photograph, picture, digital data representing any of the foregoing, orother similar representation of the external form of an animate orinanimate object, person, or thing. Some non-limiting examples of animage of the moveable input device may include data output from an imagesensor of the moveable input device, data derived from image sensordata, a picture (e.g., one or more frames) of the moveable input device,or a plurality of digital representations of the moveable input device.For example, an analysis of one or more images (e.g., data analysis) ofthe moveable input device may be used to determine the motion signalsassociated with the moveable input device, for example using anego-motion algorithm, using visual object tracking algorithm, and soforth. An image sensor may capture one or more images of the moveableinput device with each image containing data gathered by countingpixels. Thus, one image may be compared against another image to seewhere and how many pixels the two or more images contain to determine adifference which may be used to determine the motion signals associatedwith the moveable input device. This comparison of a sequence of one ormore images may be used to determine, for example, a change in position,rotation, orientation, or other similar physical movement by themoveable input device. Advantageously, in some embodiments, the analysismay take into account whether the image is a photograph or picture of akeyboard or another moveable input device to determine the motionsignals. For example, the images of the moveable input device may becaptured using at least one image sensor included in a wearable extendedreality appliance. In one example, visual tags configured to enable thedetection of the moveable input device in the images may be attached tothe moveable input device. In one example, the moveable input device mayinclude light emitters configured to emit light configured to enable thedetection of the moveable input device. The one or more light emittersmay be configured to emit visible light, infrared light, near-infraredlight, ultraviolet light, or light or electromagnetic waves of anywavelength or frequency.

By way of example, FIG. 40 depicts a scene 4000 with a moveable inputdevice 4002, optionally with at least one sensor 4008, and a secondvirtual display 4010 at a different time, position, and orientation thanthe scene 3900 in FIG. 39. In one example, moveable input device 4002may be placed on or in proximity to surface 4006. In one example,moveable input device 4002 may be the same input device as moveableinput device 3902 at different points in time, first virtual display3904 may be the same virtual display as second virtual display 4010 atdifferent points in time, and/or surface 3906 may be the same surface assurface 4006 at different points in time. Some embodiments may beconfigured wherein the motion signals associated with the moveable inputdevice 4002 (e.g., a keyboard) are determined based on an analysis ofimages of the moveable input device 4002. For example, an imaging sensorexternal to moveable input device 4002 (such as an image sensor includedin a wearable extended reality appliance presenting virtual display4010, an image sensor included in or mounted to surface 4006, an imagesensor in an environment of moveable input device 4002, and so forth)may take an image of the moveable input device 3902 at a time,orientation, and position and take a different image of the moveableinput device 4002 at a different time, orientation, and position and theprocessor may compare the differences in the two images to determine themotion signals, for example using a visual object tracking algorithm.Advantageously, a wearable extended reality appliance (e.g., smartglasses) may include a sensor (e.g., an imaging sensor) which may takean image or multiple images of the moveable input device 3902 at a time,orientation, and position and take a different image or images of themoveable input device 4002 at a different time, orientation, andposition and the processor may compare the differences in the two imagesto determine the motion signals, for example using a visual objecttracking algorithm. In other examples, the at least one sensor 4008 maybe an imaging sensor configured to capture images from an environment ofmoveable input device 3902 at different points in time, and the capturedimages may be analyzed to determine the motion signals, for exampleusing an ego-motion algorithm.

The vision or image sensor may generate signals representative of thechange in position of the one or more portions of the moveable inputdevice. By way of another example, at least one sensor 4008 may be anaccelerometer, and data captured using the accelerometer may be analyzedto determine velocities or accelerations of one or more portions of themoveable input device. A processor associated with the accelerometer maybe configured to analyze the measured velocities and/or accelerationsand determine a change in a position and/or an orientation associatedwith one or more portions of the moveable input device. Theaccelerometer may generate signals representative of the change invelocity or acceleration of the one or more portions of the moveableinput device. In another example, the one or more sensors may include alight source, such as a light-emitting diode, and/or a light detector,such as an array of photodiodes, to detect movement relative to asurface. A processor associated with a light sensor may be configured touse a beam of light to detect the presence or absence of an objectassociated with one or more portions of the moveable input device,and/or to determine the position and/or orientation of one or moreportions of the moveable input device. The light sensor may emit a lightbeam (e.g., visible or infrared) from its light-emitting element and areflective-type photoelectric sensor may be used to detect the lightbeam reflected from a target. For example, the light source may be partof an optical mouse sensor (also known as non-mechanical trackingengine) aimed towards a surface that moveable input device is placed on(such as a surface), and the motion of moveable input device may bemeasured with respect to the surface.

By way of example, FIG. 39 illustrates an exemplary scene 3900 includinga moveable input device 3902, a first virtual display 3904, and asurface 3906. Moveable input device 3902 may include at least one sensor3908 associated with the moveable input device 3902. The motion signalsof the moveable input device 3902 (e.g., a keyboard) may be determinedbased on an analysis of data captured using at least one sensor 3908(e.g., a motion sensor, image sensor, accelerometer, gyroscope, anoptical mouse sensor, etc.) associated with the moveable input device3902.

Consistent with some disclosed embodiments, the motion signals may beindicative of at least one of tilt movement, roll movement, and lateralmovement of the moveable input device. By way of example, the motionsignals may represent a tilt, a roll, or a lateral movement or change inan angle of a moveable input device relative to a horizontal, vertical,diagonal, or other plane. As used in this disclosure, tilt may refer toan act of changing physical location or position by an angle,inclination, slant, or other movement into a sloped location orposition. As used in this disclosure, roll may refer to a gyration,rotation, spin, or another type of movement in a particular direction byturning over and over on an axis. As used in this disclosure, lateralmay refer to forward-backward, sidewards, sideways, or any other type ofmovement in a single plane. For example, motion signals of the moveableinput device may be received by a processor by way of receiving valuesthat reflect a physical change in position, orientation, angle, or anyother determination of the relative position of the moveable inputdevice. By way of example, the moveable input device may tilt when themotion signals are representative of values reflecting a change of themoveable input device's position by 1°, 5°, 10°, −1°, −5°, −10°, or anyother angle compared to a horizontal, vertical, diagonal, or otherplane. By way of example, the moveable input device may roll when themotion signals are representative of values reflecting a change of themoveable input device about a fixed point, a point of rotation, by 1°,5°, 10°, −1°, −5°, −10°, or any other angle compared to the point ofrotation.

By way of example, FIG. 41 depicts a scene 4100 with a moveable inputdevice 4102 (optionally with at least one sensor 4108 associated withthe moveable input device 4102), a virtual display 4112, and a surface4106. FIG. 41 also depicts two different non-limiting examples of amovement 4114 of the moveable input device 4102 (e.g., a keyboard). Theleft part of the scene in FIG. 3 illustrates the moveable input device4102 moving 4114 (e.g., tilting) by a positive angle compared to ahorizontal plane (that is parallel to the surface 4106). The right partof the scene in FIG. 3 illustrates the moveable input device 4102 moving4114 (e.g., rotating) clockwise about a point of rotation in the centerof the moveable input device 4102. Additionally, some embodiments may beconfigured wherein the motion signals may be indicative of at least oneof tilt movement 4114, roll movement, and lateral movement of themoveable input device 4102.

Some disclosed embodiments may involve changing a size of the virtualdisplay based on the received motion signals associated with themoveable input device. As used in this context, changing the size of avirtual display may include, for example, adjusting, modifying, ortransforming a height, a width, a surface area, or any other dimensionof the virtual display. Changing the size of the virtual display mayadditionally or alternatively include, for example, adjusting,modifying, or transforming angular relationships between different sidesor features of the virtual display. In some embodiments, changing thesize may include modifying some or all dimensions of the virtualdisplay, using the same or different proportion or scaling factor. Somenon-limiting examples may include changing the size of a virtual displayby increasing or decreasing its height or width by one or moremillimeters, one or more inches, one or more feet, or one or more of anyother unit of length. Other non-limiting examples may include changingthe size of the virtual display by increasing or decreasing its relativeproportion (e.g., a ratio of two dimensions) by any type of ratio. Forexample, changing the size of the virtual display may include decreasingits height and width based on received motion signals associated withthe moveable input device when the motion signals indicate that themoveable input device is moving through a doorway or into smaller spacethan the first position or orientation. As another example, changing thesize of the virtual display may include increasing its height and widthbased on received motion signals associated with the moveable inputdevice when the motion signals indicate that the moveable input deviceis moving out of a doorway or into a larger space than a first positionor first orientation (a first position or first orientation differentthan another position or orientation). In one example, in response tofirst received motion signals (for example, corresponding to a motionlarger than a selected threshold), the size of the virtual display maybe changed, and in response to second received motion signals (forexample, corresponding to a motion smaller than the selected threshold),changing the size of the virtual display may be withheld. In anotherexample, the size of the virtual display after the change may beselected as a function of a motion corresponding to the received motionsignals (for example, a function of a magnitude of the motion, afunction of a direction of the motion, a function of a smoothness of themotion, and so forth).

By way of example, FIG. 43 depicts a change in the size of a virtualdisplay 4312 with a moveable input device 4302 (e.g., a keyboard,optionally with at least one sensor 4308 associated with the moveableinput device 4302) and the virtual display 4312. For example, the stepsmay be configured to change the size of the virtual display 4312 bydecreasing its relative proportion (e.g., a ratio of two dimensions)based on received motion signals associated with the moveable inputdevice 4302 as depicted by FIG. 43.

Some disclosed embodiments may involve outputting during a first timeperiod, first display signals to the wearable extended realityappliance. As used in this context, outputting may refer to sending asignal to a wearable extended reality appliance or to any displaydevice. Display signals may include, for example, analog or digitalelectrical signals that may cause a display device to present content inthe form of a virtual or digital representation. The virtual or digitalrepresentation may include, for example, one or more still or movingimages, text, icons, video, or any combination thereof. The graphicaldisplay may be two-dimensional, three-dimensional, holographic, or mayinclude various other types of visual characteristics. The at least oneprocessor may cause one or more analog or digital signals to begenerated and transmitted to a display device for presenting thegraphical display for viewing by a user. In some embodiments, thedisplay device may include a wearable extended reality appliance. Forexample, the at least one processor may cause one or more analog ordigital signals to be generated and transmitted to the display devicefor presenting a movie, an emoji, a video, a text, or any combinationthereof.

In some embodiments, the first display signals may be configured tocause the wearable extended reality appliance to virtually presentcontent in a first orientation. As used in this context, virtuallypresenting content may include, for example, portraying, depicting, orrendering subject matter, material, or substance that may becomputer-generated, computerized, simulated, digital, or generated usingsoftware instructions. For example, the first orientation may bedetermined as an angular position in relation to a particular positionand a particular direction (for example, of a surface, of an object, ofa coordinate system, of a surface that a moveable input device is placedon, of a coordinate system of a virtual environment, and so forth). Theat least one processor may be configured to transmit display signals tocause a display device to present text, one or more pictures, ascreenshot, a media clip, or other textual or graphical subject matterin a first orientation. The first orientation may include positioningthe displayed content at any desired angle relative to reference axes orplanes. For example, textual content may be displayed so that it isaligned horizontally with a surface of a table or a floor to enable auser to read the text. As another example, an image may be displayed sothat it is inclined at a desired angle relative to the horizontalsurface of a table of a floor. In one example, display signals (such asthe first display signals, the second display signals, etc.) may includea depiction of the virtual display corresponding to a particularorientation (such as the first orientation, the second orientation,etc.), for example, a depiction in an image, a depiction in a video, andso forth. In such example, the wearable extended reality appliance maypresent the depiction included in the display signals. In one example,display signals (such as the first display signals, the second displaysignals, etc.) may include a spatial transformation corresponding to aparticular orientation (such as the first orientation, the secondorientation, and so forth). Some non-limiting examples of such spatialtransformations may include a translation transformation, a rotationtransformation, a reflection transformation, a dilation transformation,an affine transformation, a projective transformation, and so forth. Insuch example, the spatial transformation may be applied to a depictionof the virtual display to obtain a transformed depiction of the virtualdisplay corresponding to the particular orientation, and the wearableextended reality appliance may present the transformed depiction of thevirtual display. In one example, display signals (such as the firstdisplay signals, the second display signals, etc.) may include anindication of a desired orientation for the virtual display (forexample, an indication of an angle, an indication of positions ofparticular points of the virtual display in an extended realityenvironment, and so forth). In such case, the indication of the desiredorientation (such as the first orientation, the second orientation, andso forth) may be used to transform a depiction of the virtual display toobtain a transformed depiction of the virtual display corresponding tothe desired orientation, and the wearable extended reality appliance maypresent the transformed depiction of the virtual display.

By way of example, exemplary set of instructions 4500 illustrated byFIG. 45A may include step 4504 wherein a processor outputs during afirst time period, first display signals to a wearable extended realityappliance (e.g., smart glasses), the first display signals beingconfigured to cause the wearable extended reality appliance to virtuallypresent content in a first orientation as a first virtual display 3904at a first time period.

Consistent with some disclosed embodiments, the motion signals may beconfigured to reflect physical movement of the moveable input devicerelative to a surface on which the moveable input device is placedduring the first time period. A surface may include, for example, anexterior, top, side, external, outward, or an outside part or uppermostlayer of an animate or inanimate object. Placing a moveable input deviceon a surface may include, for example, putting down, setting down,laying down, or positioning the moveable input device on the surface. Byway of example, the moveable input device may be configured to be placedon a surface and the motion signals may be configured to reflectphysical movement of the moveable input device relative to the surfaceon which the moveable input device is placed. In one example, thesurface may be the top of a desk or table. Advantageously, the motionsignals may be configured to reflect physical movement or movements(e.g., rotating, displacing, tilting) of the moveable input device(e.g., a keyboard) relative to the surface of the desk or table on whichthe moveable input device may be placed during a first time period.

By way of example, FIG. 39 illustrates reflected motion signals of thephysical orientation and/or position of the moveable input device 3902(e.g., a keyboard) relative to a surface 3906 on which the moveableinput device 3902 is placed during a first time period. In FIG. 39, themoveable input device 3902 is depicted on the front-left of the surface(e.g., a desk). In FIG. 40, the moveable input device 4002 is depictedon the front-right of the surface. In FIG. 41, the motion signalsreflect two physical movements 4114 (e.g., a tilt or a rotation) of themoveable input device 4102 relative to the surface 4106 (e.g., tilt by apositive angle compared to a horizontal plane that is parallel to thesurface 4106 or clockwise rotation about a point of rotation in thecenter of the moveable input device 4102) on which the moveable inputdevice 4102 is placed during the first time period. In FIG. 42, themoveable input device 4202 (optionally with at least one sensor 4208associated with the moveable input device 4202) is depicted on the rightside of the surface. In one example, moveable input device 4202 may bethe same input device as moveable input device 3902 and moveable inputdevice 4002 at different points in time, virtual display 4212 may be thesame virtual display as first virtual display 3904 and second virtualdisplay 4010 at different points in time, and/or surface 4206 may be thesame surface as surface 3906 and surface 4006 at different points intime.

Some disclosed embodiments may involve determining the first orientationbased on an orientation of the moveable input device before the firsttime period. An orientation of the moveable input device, as used inthis context, may include at least one of a rotation, displacement,translation, location or a position. For example, the first orientationmay be determined as an angular position in relation to a particularposition and a particular direction, the particular position and/or theparticular direction may be selected based on an orientation of themoveable input device before the first time period. In some embodiments,the first orientation may be determined during some portion of or of thefirst time period, before the first time period, and so forth. In someother examples, the first orientation may be determined based on anorientation of the moveable input device during the first time period.For example, a first orientation may be determined as an angularposition in relation to a particular position and a particulardirection, the particular position and/or the particular direction maybe selected based on an orientation of the moveable input device duringthe first time period. Likewise, the second orientation may bedetermined based on an orientation of the moveable input device beforethe second time period, during the second time period, during the firsttime period, and so forth.

By way of example, FIG. 39 illustrates sensor 3908 (e.g., a motionsensor) that may generate one or more motion signals reflecting aposition and/or orientation of moveable input device 3902 (e.g., akeyboard) during a first time period. Consistent with some disclosedembodiments, the at least one processor may further be configured todetermine a first orientation of a first virtual display 3904 based onan orientation of the moveable input device 3902 before or during thefirst time period.

Consistent with some disclosed embodiments, the at least one processormay be configured to perform steps that may include outputting during asecond time period different from the first time period, second displaysignals to the wearable extended reality appliance. A second time periodmay refer to a time period different than the first time period andsecond display signals refer to display signals different than the firstdisplay signals. The second display signal may include analog or digitalsignals different than the first display signal.

In some embodiments, the second display signals may be configured tocause the wearable extended reality appliance to virtually present thecontent in a second orientation different from the first orientation.That is, if the first orientation corresponds to one or more of aparticular rotation, displacement, translation, or particular positionrotation, displacement, translation, or particular position, the secondorientation may correspond to at least one of a rotation, displacement,translation, and/or particular position that differs in at least somerespect from the first orientation, For example, the second orientationmay be determined as an angular position in relation to a particularposition and a particular direction (for example, of a surface, of anobject, of a coordinate system, of a surface that a moveable inputdevice is placed on, of a coordinate system of a virtual environment,and so forth). The at least one processor may be configured to transmitdisplay signals to cause a display device to present text, one or morepictures, a screenshot, a media clip, or other textual or graphicalsubject matter in a second orientation. The second orientation mayinclude positioning the displayed content at any desired angle relativeto reference axes or planes. For example, textual content may bedisplayed so that it is aligned horizontally with a surface of a tableor a floor to enable a user to read the text. As another example, animage may be displayed so that it is inclined at a desired anglerelative to the horizontal surface of a table of a floor. For example,the second display signals may be configured to cause the wearableextended reality appliance to virtually present the content in thesecond orientation different from a first orientation and/or in a secondposition different from a first position.

By way of example, FIG. 40 may illustrate a moveable input device 4002(e.g., a keyboard) and a second virtual display 4010 at a second timeperiod different than first virtual display 3904 at a first time periodas shown in FIG. 39. By way of example, exemplary set of instructions4500 illustrated by FIG. 45A may include step 4506 wherein a processoroutputs during a second time period different from the first timeperiod, second display signals to a wearable extended reality appliance(e.g., smart glasses), the second display signals being configured tocause the wearable extended reality appliance to virtually present thecontent in a second orientation as a second virtual display 4010different from a first orientation as a first virtual display 3904.

Consistent with some disclosed embodiments, the motion signals may bereceived after the first time period and before the second time period.For example, the motion signals may be received at a time after a firsttime period and before a second time period. That is the motion signalsmay be received at a time that lies between the first and second timeperiods. In another example, the motion signals may be received duringthe first time period. In yet another example, the motion signals may bereceived during the second time period. In an additional example, themotion signals may be received before the first time period and thesecond time period. In one example, the first time period may be earlierthan the second time period. In another example, the first time periodmay be later than the second time period. Consistent with otherdisclosed embodiments, the at least one processor may be configured toperform steps that may include enabling the wearable extended realityappliance to receive additional motion signals during the second timeperiod, thereby enabling the wearable extended reality appliance tocontinuously adjust a virtual presentation of the content. As used inthis context, additional motion signals may refer to signals received inaddition to the other received motion signals, and may be receivedtogether with or separately from the other motion signals. Continuouslyadjusting in this context may refer to a series of adjustments (e.g.,changes calibrations, and/or adaptions) in a time period. The continuousadjustments may be discrete individual adjustments that occur in asequence, with or without intervening interruptions. In someembodiments, additional motion signals may be received during the secondtime period thus enabling the wearable extended reality appliance tocontinuously adjust a virtual presentation of content. For example,during the second time period, the wearable extended reality appliance(e.g., smart glasses) may receive additional motion signals tocontinuously adjust (e.g., change the size, change the type, change anyother aspect) of the virtual presentation of the content. By way ofanother example, continuously adjusting the virtual presentation ofcontent may take into account other physical or virtual objects in aspace (e.g., when a user walks with a moveable input device the positionand/or orientation of the content will change to avoid obstacles such aswalls, persons, and other virtual or physical objects).

In some examples, an indication that a physical object may be located ata particular location in an environment of an input device may bereceived. In some examples, image data captured using an image sensorincluded in a first wearable extended reality appliance may be received.For example, the image data may be received from the image sensor, fromthe first wearable extended reality appliance, from an intermediatedevice external to the first wearable extended reality appliance, from amemory unit, and so forth. The image data may be analyzed to detect thephysical object in the particular location in the environment. Inanother example, radar, LIDAR or sonar sensors may be used to detect thepresence of the physical object at the particular location in theenvironment. In some examples, the second orientation may be selectedbased on the physical object being located in the particular location.In one example, the second orientation may be selected such that thevirtually presented content, when presented in the second orientation,does not appear to collide with the physical object. In another example,the second orientation may be selected such that the virtually presentedcontent, when presented in the second orientation, is not hidden(completely or partly) by the physical object to a user of the wearableextended reality appliance (for example, based on the location of thewearable extended reality appliance). In one example, a ray castingalgorithm may be used to determine that the virtually presented contentpresented at a specific orientation is hidden by the physical object tothe user of the wearable extended reality appliance. Further, the seconddisplay signals may be based on the selection of the second perspectiveof the scene.

Consistent with some disclosed embodiments, the at least one processormay be configured to perform steps that may include switching betweenthe output of the first display signals and the output of the seconddisplay signals based on the received motions signals of the moveableinput device. As used in this context, switching may refer to shifting,transitioning, modifying, altering, or changing in any other way betweentwo or more items such as things, objects, positions, orientations,signals, events, or virtual displays. In some embodiments, switchingbetween an output of a first display signal or first display signals andan output of a second display signal or second display signals may bebased on received motions signals of a moveable input device (e.g., akeyboard). The received motion signals may reflect physical movement ofsome or all portions of the moveable input device such that a change inposition, place, orientation, rotation, or other similar change inposition. The at least one processor may be configured to switch or notswitch between the output of first display signals and the output ofsecond display signals. Alternatively, the received motion signals mayreflect absence of physical movement of some or all portions of themoveable input device. The at least one processor may be configured toswitch or not switch between the output of the first display signal orfirst display signals and the output of the second display signal orsecond display signals based on the received motion signals. Forexample, the received motion signals of the moveable input device mayhave a wearable extended reality device (e.g., smart glasses) switchfrom the output of the first display signal or first display signals tothe output of the second display signal or second display signals.Additionally, or alternatively, the received motion signals of themoveable input device may have the wearable extended reality deviceswitch from the output of the second display signal or second displaysignals to the output of the first display signal or first displaysignals. In one example, for example while outputting the first displaysignals, in response to a first received motion signals of the moveableinput device, the steps may include switching from the output of thefirst display signals to the output of the second display signals, andin response to a second received motion signals of the moveable inputdevice, switching from the output of the first display signals to theoutput of the second display signals may be withheld. In anotherexample, for example while outputting the second display signals, inresponse to a third received motion signals of the moveable inputdevice, the steps may include switching from the output of the seconddisplay signals to the output of the first display signals, and inresponse to a fourth received motion signals of the moveable inputdevice, switching from the output of the second display signals to theoutput of the first display signals may be withheld. In some examples,the second display signals may be determined based on the receivedmotion signals. For example, in response to one received motion signalsof the moveable input device, first values may be selected for thesecond display signals, and in response to a different received motionsignals of the moveable input device, second values may be selected forthe second display signals, the second values may differ from the firstvalues. In some examples, the second orientation may be determined basedon the received motion signals. For example, in response to one receivedmotion signals of the moveable input device, a first angle of the secondorientation may be selected, and in response to a different receivedmotion signals of the moveable input device, a second angle of thesecond orientation may be selected, the second angle may differ from thefirst angle.

By way of example, exemplary set of instructions 4500 illustrated byFIG. 45A may include step 4508 wherein a processor switches between theoutput of the first and second display signals based on received motionsignals of a moveable input device 3902.

Some disclosed embodiments may involve switching between the output ofthe first display signals and the output of the second display signalswhen the physical movement of the moveable input device is greater thanat least one threshold. As mentioned above the at least one thresholdmay refer to a reference or limit value, or level, or a range ofreference or limit values or levels. In operation, when an amount ofphysical movement of the moveable input device exceeds the at least onethreshold (or is below it, depending on a particular use case), the atleast one processor may follow a first course of action and, when theamount of physical movement of the moveable input device is below it (orabove it, depending on the particular use case), the at least oneprocessor may follow a second course of action. The value of the atleast one threshold may be predetermined or may be dynamically selectedbased on various considerations. Some non-limiting examples may includethe at least one threshold to be a predetermined value of physicalmovement by one millimeter displacement, one inch displacement, one footdisplacement, or any other amount of displacement. As another example,the at least one threshold may be a predetermined value of angularmovement of 1°, 5°, 10°, −1°, −5°, −10°, or any other angle. When theamount of movement of the moveable input device is greater than or equalto the at least one threshold, the at least one processor may switchbetween an output of a first display signal or first display signals andan output of a second display signal or second display signals, and whenthe amount of movement of the moveable input device is less than orequal to the at least one threshold, switching between an output of afirst display signal or first display signals and an output of a seconddisplay signal or second display signals may be withheld. In anotherexample, when the amount of movement of the moveable input device isless than or equal to the at least one threshold, the at least oneprocessor may switch between an output of a first display signal orfirst display signals and an output of a second display signal or seconddisplay signals.

In some embodiments, the at least one threshold may be a combination ofone or more of a tilt threshold, a roll threshold, and/or a lateralmovement threshold. In some embodiments the at least one threshold mayinclude at least one threshold of a tilt, roll, or a lateral movementthreshold. Alternatively, the at least one threshold may be configuredto require at least two or all three movements: a tilt, a roll, and alateral movement threshold. For example, when the at least one thresholdis configured to be only a tilt threshold, then if the moveable inputdevice (e.g., a keyboard) is tilted by physical movement, a processormay switch between an output of a first display signal or first displaysignals and an output of a second display signal or second displaysignals or between the output of the second display signal or seconddisplay signals and the output of the first display signal or firstdisplay signals.

In some embodiments, the at least one threshold may be selected based ona distance of the virtual display from the moveable input device duringthe first time period. As used in this context, distance may refer to alength, size, space, span, width, or any amount of space between twothings. In one example, the at least one threshold may be a function ofthe distance of the virtual display from the moveable input deviceduring the first time period. Some non-limiting examples of suchfunction may include linear function, non-linear function, polynomialfunction, monotonic function, monotonically increasing function,logarithmic function, continuous function, non-continuous function, andso forth. In one example, the function may be selected based on at leastone of the wearable extended reality appliance, a user of the wearableextended reality appliance, the moveable input device, a type of asurface that the moveable input device is placed on, one or moredimensions of a surface that the moveable input device is placed on, orwhether or not the moveable input device is placed on a surface. In someother examples, the at least one threshold may be a function of thedistance of the virtual display from the moveable input device duringthe first time period and an angle between the virtual display from themoveable input device during the first time period.

By way of example, FIG. 39 illustrates a distance 3909 between a firstdisplay 3904 and a moveable input device 3902. In some embodiments, atleast one threshold may be selected based on the distance 3909 of thefirst virtual display 3904 from the moveable input device 3902 (e.g., akeyboard) during a first time period.

In some embodiments, the at least one threshold may be selected based onan orientation of the virtual display from the moveable input deviceduring the first time period. For example, the at least one thresholdmay be selected based on an orientation (e.g., upright, upside down,parallel, perpendicular, or any other orientation) and/or a position(e.g., a middle, a corner, a side, a face, or any other position) of thevirtual display with respect to a moveable input device (e.g., akeyboard) during a first time period. For example, the at least onethreshold may be set to a higher value (e.g., one orientation and/orposition of the moveable input device may require a larger thresholdlevel than a different orientation and/or position) based on theorientation and/or position of the virtual display being upside downcompared to upright or the virtual display being in the corner of asurface (e.g., a desk) compared to being in the middle of the desk.Alternatively, the at least one threshold may be set to a lower value(e.g., one orientation and/or position of the moveable input device mayrequire a smaller threshold level than a different orientation and/orposition) based on the orientation and/or position of the virtualdisplay being upright compared to upside down or the virtual displaybeing in the middle compared to being in the corner of the surface.

By way of example, FIG. 42 illustrates scene 4200 with a virtual display4212 at an orientation and position at a first time period. In someembodiments, at least one threshold may be selected based on anorientation and/or position of the virtual display 4212 (e.g., rotated90° and at a side of a surface 4206) during the first time period.

In some embodiments, at least one threshold is selected based on a typeof the content. In some examples, content type may include images, text,symbols, codes, mixed media, or any other information presented,regardless of form. By way of example, the at least one threshold may beassigned a higher value when the content is textual as opposed topictorial. As another example, the at least one threshold may beassigned a lower value when the content is a multi-media type content asopposed to being textual or pictorial. In some examples, content typemay include private and public content. By way of example, the at leastone threshold may be assigned a higher value when the content is publicas opposed to private.

Consistent with some disclosed embodiments, the moveable input devicemay be configured to be placed on a surface and a value of the at leastone threshold may be based on a type of the surface. As used in thiscontext, type of surface may refer to a kind, variety, classification,dimensions, or other category having common characteristics. Somenon-limiting examples of types of surfaces may include a table, a desk,a bed, a floor, a counter, a wall, and any other object having asurface. Other examples may include types of surfaces that are temporary(e.g., a dining table) and/or fixed (e.g., an office desk). For example,the moveable input device (e.g., a keyboard) may be configured to beplaced on the surface of an office desk and a higher value of the atleast one threshold may be based on the type of surface (e.g., theoffice desk). Alternatively, the moveable input device may be configuredto be placed on the surface of a dining table and a lower value of theat least one threshold may be based on the type of surface (e.g., thedining table). For example, a higher value for the at least onethreshold may be associated with the office desk, a fixed surface,because of the desk's inherent qualities (e.g., a greater shear force orgreater resistance for physical movement of the moveable input devicecompared to a temporary surface). Another example may include a lowervalue for the at least one threshold associated with the dining table, atemporary surface, because of the dining table's inherent qualities(e.g., a lower shear force or lower resistance for physical movement ofthe moveable input device compared to a fixed surface). Additionally, oralternatively, the at least one threshold, when based on a type ofsurface, may be configured to have a lower or higher value because oftemporariness of using the moveable input device on that particularsurface (e.g., the moveable input device may be used more frequently onan office desk compared to a dining table). In one example, the at leastone threshold may be a function of at least one dimension of thesurface. Some non-limiting examples of such function may include linearfunction, non-linear function, polynomial function, monotonic function,monotonically increasing function, logarithmic function, continuousfunction, non-continuous function, and so forth.

By way of example, FIG. 39 illustrates a first virtual display 3904, amoveable input device 3902, and a surface 3906. In one example, themoveable input device 3902 may be placed on the surface 3906 (e.g., asurface of a desk) and a value of the at least one threshold may bebased on the type of the surface. For example, the moveable input device3902 may be placed on a temporary surface 3906 (e.g., a dining table) ora fixed surface 3906 (e.g., an office desk) and a threshold value may beassigned based on the type of surface.

Consistent with some disclosed embodiments, the wearable extendedreality appliance may be configured to be paired with multiple moveableinput devices and the first orientation may be determined based ondefault virtual display configurations associated with one of themultiple moveable input devices paired with the wearable extendedreality appliance. As used in this context, multiple moveable inputdevices may refer to two, three, more than one, more than two, more thanthree, numerous, various, or several moveable input devices as discussedherein. Some non-limiting examples may include a wearable extendedreality appliance paired with two moveable input devices, three moveableinput devices, four moveable input devices, or any number of moveableinput devices. When a wearable extended reality appliance is paired withmore than one moveable input device, a first orientation of the virtualcontent may be determined based on a default virtual displayconfiguration associated with any one of or more than one of themoveable input devices paired with the wearable extended realityappliance. For example, the wearable extended reality appliance (e.g.,smart glasses) may be configured to be paired with multiple moveableinput devices (e.g., a keyboard and a mouse) and the first orientationmay be determined based on default virtual display configurationsassociated with either the keyboard or the mouse paired with thewearable extended reality appliance. Advantageously, different moveableinput devices may be associated with different default virtual displayconfigurations. For example, the default virtual display configurationfor a mouse may be set with a virtual display on a user's right-handside, with any type of color and/or font scheme for the virtual content,with a smaller display size, or any other predetermined configurationassociated with a mouse. Other examples may include the default virtualdisplay configuration for a keyboard which may be set with a virtualdisplay directly in front of the user's view, with any type of colorand/or font scheme for the virtual content, an automated introductoryvoice message (e.g., one welcoming the user), with a larger displaysize, or any other predetermined configuration or configurationsassociated with a keyboard.

By way of example, FIG. 44 illustrates a scene 4400 with a moveableinput device 3902 with at least one sensor 4408 associated with themoveable input device 4402, a virtual display 4412, a surface 4406, asecond moveable input device 4416 at a time period, and a virtualdisplay 4419 associated with the second moveable input device 4416. Someembodiments may be configured wherein a wearable extended realityappliance (e.g., smart glasses) may be paired with multiple moveableinput devices (the moveable input device 4402 (e.g., a keyboard) and thesecond moveable input device 4416 (e.g., a mouse)) and the firstorientation may be determined based on default virtual display 4112configurations associated with one of the multiple moveable inputdevices paired with the wearable extended reality appliance.

Consistent with some disclosed embodiments, content may be a virtualdisplay configured to enable visual presentation of textual inputentered using the moveable input device. As used in this context,textual input may refer to any word, character, or string of charactersentered into a system by a user or other device. Some non-limitingexamples of textual input may include “HELLO,” “HI,” “A,” “Hello World,”“ABC,” or any other combination of letters, words, or punctuation marks.For example, a virtual display may be configured to display textualinput (e.g., “HELLO,” “HI,” “A,” “Hello World.”) entered using amoveable input device (e.g., a keyboard), for example in a userinterface, in a text editor, and so forth.

Some disclosed embodiments may involve providing a visual indicationoutside the virtual display of textual input entered using the moveableinput device when the virtual display is outside a field of view of thewearable extended reality appliance. As used in this context, visualindication may refer to any visual sign, indicator, mark, signal, orother sign or piece of information that indicates something. A field ofview may refer to a line of sight, a line of vision, a peripheral field,a peripheral vision, or extent of an observable world that is seen atany given moment. Some non-limiting examples of a field of view mayinclude a 210-degree forward-facing horizontal arc, 150 degrees, 60degrees, 45 degrees, and so forth. Some non-limiting examples of avisual indication may include a “!,” “WARNING,” “Outside field ofview,”, a blinking light, a graphical symbol, or any other similartextual or graphical symbol, picture, video, or word. For example, auser may use a moveable input device (e.g., a keyboard) and entertextual input and when a virtual display (e.g., a screen) is outside afield of view (e.g., 210-degree forward-facing horizontal arc) of awearable extended reality appliance (e.g., smart glasses), the at leastone processor may cause the wearable extended reality appliance todisplay a symbol “!” as a visual indication outside the virtual display.In one example, when the moveable input device is in the field of viewof the wearable extended reality appliance, the at least one processormay cause the wearable extended reality appliance to provide the visualindication over the moveable input device. In one example, when themoveable input device is in the field of view of the wearable extendedreality appliance and the virtual display is outside the field of viewof the wearable extended reality appliance, the at least one processormay cause the wearable extended reality appliance to provide the visualindication over the moveable input device. In one example, when both themoveable input device and the virtual display are outside the field ofview of the wearable extended reality appliance, the at least oneprocessor may cause the wearable extended reality appliance to providethe visual indication outside the virtual display and not over themoveable input device.

FIG. 45A illustrates an exemplary method 4500 for integrating a moveableinput device with a virtual display projected via a wearable extendedreality appliance. Method 4500 may be performed by one or moreprocessing devices (e.g., 360, 460, or 560) associated with input unit202 (see FIG. 3), XR unit 204 (see FIG. 4), and/or remote processingunit 208 (see FIG. 5). The steps of the disclosed method 4500 may bemodified in any manner, including by reordering steps and/or insertingor deleting steps. Method 4500 may include a step 4502 of receivingmotion signals associated with the moveable input device. The motionsignals may reflect physical movement of the moveable input device.Method 4500 may include a step 4504 of outputting during a first timeperiod, first display signals to the wearable extended realityappliance. The first display signals may be configured to cause thewearable extended reality appliance to virtually present content in afirst orientation. Method 4500 may include a step 4506 of outputtingduring a second time period, second display signals to the wearableextended reality appliance. The second display signals may be configuredto cause the wearable extended reality appliance to virtually presentcontent in a second orientation different from the first orientation.Method 4500 may include a step 4508 of switching between the output ofthe first and second display signals based on the received motionsignals of the moveable input device.

FIG. 45B illustrates another exemplary process for integrating amoveable input device with a virtual display projected via a wearableextended reality appliance. Method 4550 may be performed by one or moreprocessing devices (e.g., 360, 460, or 560) associated with input unit202 (see FIG. 3), XR unit 204 (see FIG. 4), and/or remote processingunit 208 (see FIG. 5). The steps of the disclosed method 4550 may bemodified in any manner, including by reordering steps and/or insertingor deleting steps. Method 4550 may include a step 4552 of receivingmotion signals associated with the moveable input device. The motionsignals may reflect physical movement of the moveable input device.Method 4550 may include a step 4554 of outputting during a first timeperiod, first display signals to the wearable extended realityappliance. The first display signals may be configured to cause thewearable extended reality appliance to virtually present content in afirst orientation. Method 4550 may include a step 4556 of outputtingduring a second time period, second display signals to the wearableextended reality appliance. The second display signals may be configuredto cause the wearable extended reality appliance to virtually presentcontent in a second orientation different from the first orientation.Method 4550 may include a step 4560, if the moveable input device'smovement is greater than the at least one threshold, then step 4558, maybe performed wherein the step switches between the output of the firstand second display signals based on the received motion signals of themoveable input device.

While ordinary extended reality appliances may present virtual objectsto users, it may be desirable to extend a physical keyboard to asurrounding surface using a wearable extended reality appliance. Thistype of extended display may allow a user to interact with the keyboardbeyond its physical limits. Additionally, maintaining a spatialorientation between the physical keyboard and its corresponding virtualkeyboard may allow a user to continue interaction with the virtualkeyboard when the user needs to move the physical keyboard withoutconfusing the user. The following disclosure describes various systems,methods, and non-transitory computer readable media for virtuallyextending a physical keyboard.

A physical keyboard may include one or a combination of QWERTY keyboards(e.g., mechanical keyboards, membrane keyboards, flexible keyboards) orother types of computer keyboards (e.g., Dvorak and Colemak), a chordedkeyboard, a wireless keyboard, a keypad, a key-based control panel, oranother array of control keys, vision input devices, or any othermechanism for inputting text that is provided in physical form. Aphysical form may include an object that is tangible, palpable,concrete, or in any other manner has a material, as opposed to a virtualor transitory, existence.

Virtually extending a physical keyboard may involve copying,duplicating, continuing, developing, enhancing, augmenting,supplementing, connecting, associating, attaching, relating, affixing,correlating, coupling, tagging, interfacing, or in any other way linkingthe physical keyboard to a virtual environment. A virtual environmentmay include a simulated and non-physical environment that provides auser with the perception of being present in an environment that doesnot physically exist. For example, virtually extending a physicalkeyboard may involve duplicating the form of a physical keyboard in avirtual environment. As another example, virtually extending a physicalkeyboard may involve associating an object in the virtual environmentwith the physical keyboard.

Some disclosed embodiments may include receiving image data from animage sensor associated with a wearable extended reality appliance, theimage data representing a keyboard placed on a surface. An image sensormay be included in any of the devices or systems in the presentdisclosure and may be any device capable of detecting and convertingoptical signals in the near-infrared, infrared, visible, and ultravioletspectrums into electrical signals. Examples of image sensors may includedigital cameras, phone cameras, semiconductor Charge-Coupled Devices(CCDs), active pixel sensors in Complementary Metal-Oxide-Semiconductor(CMOS), or N-type metal-oxide-semiconductor (NMOS, Live MOS). Theelectrical signals may be used to generate image data. Consistent withthe present disclosure, the image data may include pixel data streams,digital images, digital video streams, data derived from capturedimages, and data that may be used to construct one or more 3D images, asequence of 3D images, 3D videos, or a virtual 3D representation.

Consistent with one aspect of the disclosure, the extended realityappliance may be a wearable device, such as a head-mounted deviceincluding, for example, smart glasses, smart contact lens, headsets orany other device worn by a human for purposes of presenting an extendedreality to the human. Other wearable extended reality appliances mayinclude a holographic projector or any other device or system capable ofproviding an augmented reality (AR), virtual reality (VR), mixed reality(MR), or any immersive experience. Typical components of wearableextended reality appliances may include at least one of: a stereoscopichead-mounted display, a stereoscopic head-mounted sound system,head-motion tracking sensors (such as gyroscopes, accelerometers,magnetometers, image sensors, structured light sensors, etc.), headmounted projectors, eye-tracking sensors, and any other device that maybe coupled to a wearable device.

A surface may include an area, exterior, side, top, plane, face, shell,cover, or any other outside part or upper boundary of an object or body.In one example, a surface may be the top of a table on which thekeyboard is placed. In another example, a surface may be a lap of a userof a keyboard, such that the keyboard is placed on the lap of the user.The surface may encompass an area that is bigger than, smaller than, orequal to an area of the keyboard. In one example, the surface may be atop of a table. In some embodiments, the top of the table may be smallerthan the keyboard. In other embodiments, the top of the table may belarger than the keyboard. In yet other embodiments, the top of the tablemay be the same size as the keyboard. The surface may include a single,continuous surface or any combination of a plurality of surfaces. Forexample, the surface may include a top of a first table that adjoins atop of a second table. As an example, in FIG. 47, the surface 4710 isdepicted as the top of a table.

Some disclosed embodiments may include determining that the keyboard ispaired with the wearable extended reality appliance. Pairing thekeyboard with the wearable extended reality appliance may involvecoupling, uniting, linking, affixing, conjoining, interfacing, joining,or in any other way connecting the keyboard to the wearable extendedreality appliance. The pairing between the keyboard and the wearableextended reality appliance may include any or a combination of one ormore of a wired or wireless pairing. A pairing may include either awired pairing or a wireless pairing. A wired pairing may utilize coaxialcables, Ethernet, or any other channel that transmits information over awired connection between the keyboard and the wearable extended realityappliance. A wireless pairing may utilize WiFi, Bluetooth™, or any otherchannel that transmits information without a wired connection betweenthe keyboard and the wearable extended reality appliance. In anotherexample, the pairing may utilize indirect communication between thekeyboard and the wearable extended reality appliance, for examplethrough an additional computerized system and/or a communicationnetwork. The additional computerized system may control an extendedreality environment presented through the wearable extended realityappliance. In yet another example, the pairing of the keyboard with thewearable extended reality appliance may include a configuration thatcauses text typed using the keyboard to be presented via the wearableextended reality appliance, for example in a virtual display presentedvia the wearable extended reality appliance, for example in a texteditor and/or in a user interface presented on the virtual display.

Determining that the keyboard is paired with the wearable extendedreality appliance may involve detecting signals from a proximity,pressure, light, ultrasonic, position, photoelectric, motion, force,electrical, contact, non-contact, or any other type of sensor. In someexamples, determining that the keyboard is paired with the wearableextended reality appliance may be based on a detection of the keyboardin an image captured by an image sensor included in the wearableextended reality appliance. In some examples, determining that thekeyboard is paired with the wearable extended reality appliance may bebased on a detection of a visual code associated with the keyboard in animage captured by an image sensor included in the wearable extendedreality appliance. In some examples, determining that the keyboard ispaired with the wearable extended reality appliance may be based on adetection of a light emitted by a light emitter included in the keyboardin a data captured by a sensor included in the wearable extended realityappliance.

Some disclosed embodiments may include receiving an input for causing adisplay of a virtual controller in conjunction with the keyboard. Aninput may involve a user input or a sensor input or a data input. Insome embodiments, the input may be a user input, or one in which a userenters an input through an input device. The input device may includeany physical device configured to receive input from a user or anenvironment of the user, and to provide the data to a computationaldevice. The data provided to the computational device may be in adigital format and/or in an analog format. In one embodiment, the inputdevice may store the input received from the user in a memory deviceaccessible by a processing device, and the processing device may accessthe stored data for analysis. In another embodiment, the input devicemay provide the data directly to a processing device, for example, overa bus or over another communication system configured to transfer datafrom the input device to the processing device. In some examples, theinput received by the input device may include key presses, tactileinput data, motion data, position data, direction data, or any otherdata for supply for computation. Some examples of the input device mayinclude a button, a key, a keyboard, a computer mouse, a touchpad, atouchscreen, a joystick, or another mechanism from which input may bereceived. In other embodiments, the input may be a sensor input, or onein which sensor data is used as an input to the system. The sensor datamay include data from any or a combination of a position sensor,pressure sensor, temperature sensor, force sensor, vibration sensor,photoelectric sensor, or any other type of device that can measure anyattribute of an environment. In some other examples, the input forcausing the display of the virtual controller in conjunction with thekeyboard may be received from an external device, may be received from amemory unit (for example, from a configuration file), and so forth.

A virtual controller may include a simulated non-physical controllerwhich provides to the user the perception of interacting with acontroller that is not physically present. The virtual controller mayinclude any one or a combination of a button, a key, a keyboard, acomputer mouse, a touchpad, a trackpad, a touchscreen, a joystick, abutton, a slider, a dial, a keypad, a numeric keypad, or anothermechanism that may be manipulated by a user in a virtual environment. Insome embodiments, the virtual controller may be of the same form as thephysical keyboard. In other embodiments, the virtual controller may beof a different form from the physical keyboard. FIG. 46 shows an exampleof a virtual controller, consistent with some embodiments of the presentdisclosure. The virtual controller may include a virtual keyboard 4610,a virtual volume bar 4612, and a virtual touch pad 4614.

Some disclosed embodiments may include displaying, via the wearableextended reality appliance, the virtual controller in a first locationon the surface, wherein in the first location, the virtual controllerhas an original spatial orientation relative to the keyboard. A spatialorientation may include a location, position, distance, angle,placement, alignment, inclination, or any other indication of adirection of an object relative to another object. For example, FIG. 47shows a virtual controller 4716 displayed in a first location 4718 on asurface 4710, such that the virtual controller has an original spatialorientation 4720 relative to the keyboard 4712 that is placed on a firstkeyboard location 4714 on the surface 4710.

Some disclosed embodiments may include detecting a movement of thekeyboard to a different location on the surface. Detecting the movementof the keyboard may involve detecting signals from a sensor that sensesproximity, pressure, light, ultrasonic, position, photoelectric, motion,force, electric fields, contact, non-contact, or any other detectablecharacteristic. In one example, detecting the movement of the keyboardmay include detecting the movement by analyzing images using a visualobject tracking algorithms (for example, by analyzing images capturedusing an image sensor included in the wearable extended realityenvironment). The detection of the movement of the keyboard may beimplemented using other sensors and/or techniques, as described herein.In some embodiments, the detection may include detecting any amount ofmotion of the keyboard on the surface. In other embodiments, thedetection may include detecting a movement of the keyboard to adifferent location beyond a threshold movement amount.

Some disclosed embodiments may include, in response to the detectedmovement of the keyboard, presenting the virtual controller in a secondlocation on the surface, wherein in the second location, a subsequentspatial orientation of the virtual controller relative to the keyboardcorresponds to the original spatial orientation. A subsequent spatialorientation that corresponds to the original spatial orientation mayinclude a spatial orientation that shares the same or a similarlocation, position, angle, placement, alignment, inclination, or anyother indication of a direction of the virtual controller relative tothe keyboard. For example, FIG. 47 shows a virtual controller 4716displayed in a first location 4718 on a surface 4710, such that thevirtual controller has an original spatial orientation 4720 relative tothe keyboard 4712 that is placed on a first keyboard location 4714 onthe surface 4710. When the keyboard 4712 is moved to a differentkeyboard location 4722 on the surface 4710, the virtual controller 4716is presented in a second location 4724 on the surface 4710, wherein asubsequent spatial orientation 4720 of the virtual controller 4716relative to the keyboard 4712 corresponds to the original spatialorientation 4720. In this example, the keyboard 4712 was moved to adifferent keyboard location 4722 by horizontal movement.

As another example, FIG. 48 shows a virtual controller 4816 displayed ina first location 4818 on a surface 4810, such that the virtualcontroller has an original spatial orientation 4820 relative to thekeyboard 4812 that is placed on a first keyboard location 4814 on thesurface 4810. When the keyboard 4812 is moved to a different keyboardlocation 4822 on the surface 4710, the virtual controller 4816 ispresented in a second location 4824 on the surface 4810, wherein asubsequent spatial orientation 4820 of the virtual controller 4816relative to the keyboard 4812 corresponds to the original spatialorientation 4820. In this example, the keyboard 4812 was moved to adifferent keyboard location 4822 by rotational movement.

Furthermore, the portion of the virtual controller 4816 that moves withthe keyboard 4812 may depend on the portions of the virtual controller4816 that are docked to the keyboard 4812. Docking may involveanchoring, joining, uniting, connecting, or in any other way linking aportion of the virtual controller with the keyboard. For example, thevirtual controller shown in FIG. 46 may include a virtual keyboard 4610,a virtual volume bar 4612, and a virtual touch pad 4614, all of whichare docked to a keyboard. In this example, the virtual keyboard 4610, avirtual volume bar 4612, and a virtual touch pad 4614 will move inresponse to the movement of the keyboard. However, an undocked object,such as an object next to the virtual volume bar 4612 may not move inresponse to the movement of the keyboard.

In some examples, before the detection of the movement of the keyboardto the different location on the surface, detecting a hand movement of aparticular movement type at the first location may trigger a particularaction while detecting a hand movement of the particular movement typeat the second location may not trigger the particular action. Further,after the detection of the movement of the keyboard to the differentlocation on the surface, detecting a hand movement of the particularmovement type at the second location may trigger the particular actionwhile detecting a hand movement of the particular movement type at thefirst location may not trigger the particular action. In one example,the particular action may include changing a presentation parameterassociated with a user interface element based on the detected handmovement, for example as described below. In one example, the particularaction may include changing a location of a virtual cursor based on thedetected hand movement, for example as described below.

In some embodiments, the virtual controller may be a user interfaceelement, and some disclosed embodiments may include detecting a handmovement at the second location, and changing a presentation parameterassociated with the user interface element based on the detected handmovement. A user interface element may include a volume bar, a touchpad, or any other point of interaction between a user and the virtualcontroller. A hand movement may include a touch of the hand with thesurface, a click of the hand on the surface, a double click on thesurface, a drag of a finger on the surface, a drag of two fingers on thesurface, and any other interaction of any finger or other portion of thehand with the surface. A presentation parameter may include orientation,size, location, position, angle, color, dimension, volume, or any otherperceivable condition of a presented object. In some embodiments, thechange to the presentation parameter may be based on the type of thehand movement and/or on a parameter of the hand movement (such asduration, distance, etc.). For example, in response to a user draggingtheir finger vertically upward on a virtual volume bar, the volume ofthe presented scene may be increased, and the volume may be increased inproportion to the amount the user vertically drags their finger on thevirtual bar. In other embodiments, the presentation parameter may beselected from a plurality of alternative presentation parameters basedon the type of the hand movement. For example, the user interfaceelement may be a touch pad, and single-finger hand movements may be usedto adjust a size of a portion of a presented scene, while double-fingermovements may be used to adjust an orientation of the portion of thepresented scene.

In some embodiments, the virtual controller may be a virtual track padand some disclosed embodiments may include detecting a hand movement inthe second location, and changing a location of a virtual cursor basedon the detected hand movement. A virtual track pad may include any typeof virtual interface that translates the position and/or motion of anyportion of a hand of a user in order to change the location the virtualcursor. A hand movement may include a touch of the hand with thesurface, a click of the hand on the surface, a double click on thesurface, a drag of a finger on the surface, a drag of two fingers on thesurface, and any other interaction of any finger or other portion of thehand with the surface. For example, a user may drag a finger from leftto right in the second location of the surface, and the virtual cursormay be moved from left to right on the virtual track pad in response tothis detected hand movement. As another example, a user may draw acircle with a finger in a clockwise motion in the second location of thesurface, and the virtual cursor may be moved in a clockwise manner onthe virtual track pad in response to this detected hand movement.

In some embodiments, the received input may include image data from theimage sensor associated with the wearable extended reality appliance,and some disclosed embodiments may include determining from the imagedata values characterizing the original spatial orientation of thevirtual controller relative to the keyboard. For example, the image datamay be analyzed to select the first location on the surface, and therebydetermining the values characterizing the original spatial orientationof the virtual controller relative to the keyboard. In another example,the image data may be analyzed to select the values characterizing theoriginal spatial orientation of the virtual controller relative to thekeyboard, and the first location on the surface may be selected based onthe selected values. In one example, in response to a first image data,first values characterizing the original spatial orientation of thevirtual controller relative to the keyboard may be selected, and inresponse to a second image data, second values characterizing theoriginal spatial orientation of the virtual controller relative to thekeyboard may be selected, the second values may differ from the firstvalues. In one example, a convolution of at least part of the image datamay be calculated to obtain a result value of the calculatedconvolution. Further, in response to a first result value of thecalculated convolution, first values characterizing the original spatialorientation of the virtual controller relative to the keyboard may beselected, and in response to a second result value of the calculatedconvolution, second values characterizing the original spatialorientation of the virtual controller relative to the keyboard may beselected, the second values may differ from the first values. In anotherexample, the values characterizing the original spatial orientation ofthe virtual controller relative to the keyboard may be a function of theresult value of the calculated convolution. In some examples, a machinelearning model may be trained using training examples to select valuescharacterizing spatial orientations for virtual controllers relative tophysical keyboards from images of the physical keyboards. An example ofsuch training example may include an image of a sample physical keyboardand an indication of a type of a sample virtual controller, togetherwith a label indicating a value characterizing a desired spatialorientation for the sample virtual controller relative to the samplephysical keyboard. The trained machine learning model may be used toanalyze the image data received from the image sensor associated withthe wearable extended reality appliance to determine the valuescharacterizing the original spatial orientation of the virtualcontroller relative to the keyboard. In some examples, a machinelearning model may be trained using training examples to selectlocations for virtual controllers based on images of physical keyboards.An example of such training example may include a sample image of asample physical keyboard and an indication of a type of a sample virtualcontroller, together with a label indicating a desired location for thesample virtual controller relative to the sample physical keyboard. Thetrained machine learning model may be used to analyze the image datareceived from the image sensor associated with the wearable extendedreality appliance to determine the first location on the surface. Somedisclosed embodiments may further include the values characterizing adistance between the virtual controller and the keyboard. In otherexamples, the values may characterize an angle between the virtualcontroller and the keyboard, an elevation between the virtual controllerand the keyboard, a size ration (or different) between the virtualcontroller and the keyboard, and so forth.

Some disclosed embodiments may include using the received input todetermine at least one of at least one of distance of the virtualcontroller from the keyboard, an angular orientation of the virtualcontroller relative to the keyboard, a side of the keyboard on which thevirtual controller is located, or a size of the virtual controller. Inone example, in response to a first input, a first distance of thevirtual controller from the keyboard may be selected, and in response toa second input, a second distance of the virtual controller from thekeyboard may be selected, the second distance may differ from the firstdistance. In one example, in response to a first input, a first angularorientation of the virtual controller relative to the keyboard may beselected, and in response to a second input, a second angularorientation of the virtual controller relative to the keyboard may beselected, the second angular orientation may differ from the firstangular orientation. In one example, in response to a first input, afirst side of the keyboard to locate the virtual controller at may beselected, and in response to a second input, a second side of thekeyboard to locate the virtual controller may be selected, the secondside may differ from the first side. In one example, in response to afirst input, a first size for the virtual controller may be selected,and in response to a second input, a second size for the virtualcontroller may be selected, the second size may differ from the firstsize. In some examples, a machine learning model may be trained usingtraining examples to select properties for placement of virtualcontrollers (such as distance from a keyboard, an orientation relativeto the keyboard, a side of the keyboard to place the virtual controllerat, a size of the virtual controller, etc.) from inputs. An example ofsuch training example may include a sample input and an indication of atype of a sample virtual controller, together with a label indicatingdesired properties for a placement of the sample virtual controller. Thetrained machine learning model may be used to analyze the input anddetermine the at least one of the distance of the virtual controllerfrom the keyboard, the angular orientation of the virtual controllerrelative to the keyboard, the side of the keyboard on which the virtualcontroller is to be located, or the size of the virtual controller.

In some embodiments, the keyboard may include a detector and detectingthe movement of the keyboard may be based on an output of the detector.A detector may include a passive infrared sensor, a microwave sensor, anarea reflective sensor, an ultrasonic sensor, a vibration sensor, or anyother type of device that may be used to measure a movement of anobject. An output of the detector used to detect the movement of thekeyboard may include a temperature, reflection, distance, vibration, orany other indication of a movement of an object. For example, thedetector may be an ultrasonic motion sensor, and the movement of thekeyboard may be detected based on reflections that are measured viapulses of ultrasonic waves. In one example, the detector may be anindoor positioning sensor, and the output of the detector may be aposition of the keyboard. In another example, the detector may includean image sensor, images captured using the image sensor may be analyzedusing an egomotion algorithm to detect movement of the keyboard. In yetanother example, the detector may include an optical mouse sensor (alsoknown as non-mechanical tracking engine) aimed towards a surface thatthe keyboard is placed on, and the output of the detector may beindicative of the movement of the keyboard with respect to the surface.

Some disclosed embodiments may include detecting the movement of thekeyboard based on data obtained from the image sensor associated withthe wearable extended reality appliance. For example, the image sensormay be a complementary metal oxide semiconductor (CMOS) sensor designedto measure the distance to an object via the time-of-flight (TOF)method, and the image data from the CMOS sensor may be used to determinea distance between the virtual controller and the keyboard using the TOFmethod. In this example, the measured distance between the virtualcontroller and the keyboard using the TOF method may be used to detect amovement of the keyboard. In one example, the data obtained from theimage sensor associated with the wearable extended reality appliance maybe analyzed using a visual object tracking algorithm to detect movementof the keyboard.

In some embodiments, the wearable extended reality appliance may beconfigured to pair with a plurality of differing keyboards, and theseembodiments may include receiving a keyboard selection, choosing thevirtual controller from among a plurality of choices based on thereceived keyboard selection, and displaying the virtual controller basedon the keyboard selection. For example, a data-structure associatingdifferent keyboards with different selection of virtual controller maybe accessed based on the received keyboard selection to choose thevirtual controller from among the plurality of choices. In someembodiments, displaying the virtual controller based on the keyboardselection may further include accessing data indicative of a pluralityof surfaces, identifying a specific surface out of the plurality ofsurfaces on which the keyboard is placed, and selecting which virtualcontroller to display based on the identified surface. For example, adata-structure associating different surfaces with different selectionof virtual controller may be accessed based on the specific surface toselect which virtual controller to display. A plurality of differingkeyboards may include keyboards that are distinct from one another bytype, number, or size. A keyboard selection may include a selection of akeyboard based on a user input or an automatic input.

A user input may include a click, tap, swipe, or any interaction withany element of a system usable with some disclosed embodiments in orderto select a keyboard from a plurality of differing keyboards. Forexample, a user may be presented with a choice between a keyboard with30 keys and a keyboard with 50 keys, and the user may select thekeyboard with 30 keys by pressing a button on the keyboard with 30 keysto pair the keyboard with the wearable extended reality appliance. Asanother example, with the same choice of keyboards, a keyboard selectionmay be accomplished by the user clicking on virtual image of theselected keyboard presented by the wearable extended reality appliance.

An automatic input may include an automated selection of a keyboard froma plurality of differing keyboards based on a compatibility of a type ofkeyboard with a type of interaction or virtual controller desired. Forexample, a user may be presented with a choice between a keyboardwithout a volume bar and a keyboard with a volume bar, and the wearableextended reality appliance may automatically select the keyboard withthe volume bar for pairing when a scene presented by the wearableextended reality appliance includes sound that a user may want toadjust.

In some embodiments, a keyboard selection may be based on a combinationof a user input and an automatic input, such that a selection of akeyboard from a plurality of differing keyboards is based on both anautomatic selection a keyboard based on a compatibility of a type ofkeyboard with a type of interaction or virtual controller desired andconfirmation of the selection based on user input. For example, a usermay be presented with a choice between a keyboard without a volume barand a keyboard with a volume bar, and the wearable extended realityappliance may automatically select the keyboard with the volume bar forpairing when a scene presented by the wearable extended realityappliance includes sound that a user may want to adjust. The user maythen confirm this automatic selection by clicking on a virtualconfirmation button presented by the wearable extended realityappliance.

Some disclosed embodiments may include analyzing the image data todetermine that a surface area associated with the second location isdefect-free; in response to the determination that the surface areaassociated with the second location is defect-free, causing the wearableextended reality appliance to virtually present the virtual controllerin the second location; analyzing the image data to determine that thesurface area associated with the second location includes a defect; andin response to the determination that the surface area associated withthe second location includes a defect, causing the wearable extendedreality appliance to perform an action for avoiding presentation of thevirtual controller in the second location. A defect may include a crack,deformity, flaw, irregularity, blot, discoloration, kink, mark, patch,speck, stain, taint, rough spot, weak spot, or any other imperfection onthe surface. Adjusting the presentation of the virtual controller basedon whether or not the surface area associated with the second locationis defect-free may be desirable to ensure that the presentation of thevirtual controller is not deformed or otherwise flawed because of adefect in the background. For example, the image data may be analyzed todetermine that a surface area associated with the second location isfree of stains. If it is determined that the surface area associatedwith the second location is free of stains, the wearable extendedreality appliance may virtually present the virtual controller in thesecond location. However, if it is determined that the surface areaassociated with the second location contains a stain, the wearableextended reality appliance may perform an action for avoidingpresentation of the virtual controller in the second location.Additionally, the determination of whether a surface area associatedwith the second location is defect-free may include a tolerance fordefects within a certain threshold dimension. For example, a stain witha width less than 5 millimeters may be classified as defect-free,allowing the wearable extended reality appliance to virtually presentthe virtual controller in the second location, while a stain with awidth greater than or equal to 5 millimeters may be classified as adefect, allowing the wearable extended reality appliance may perform anaction for avoiding presentation of the virtual controller in the secondlocation.

In some embodiments, the action for avoiding presentation of the virtualcontroller in the second location includes virtually presenting thevirtual controller on another surface area in a third location proximatethe second location. Another surface area in a third location proximatethe second location may include any area in a location that at leastpartially avoids presentation of the virtual controller in the secondlocation. For example, when a defect is detected in the surface areaassociated with the second location, the wearable extended realityappliance may virtually present the virtual controller to the right orleft of the defect.

In some embodiments, the action for avoiding presentation of the virtualcontroller in the second location may include providing a notice via thewearable extended reality appliance, the notice indicating that thesecond location is unsuitable for displaying the virtual controller. Thenotice may include any or a combination of an alarm, a beep, a text box,a color change, or any other audio, visual, haptic, or any type ofsensory indication perceivable by a user of the wearable extendedreality appliance. For example, when a defect is detected in the surfacearea associated with the second location, the wearable extended realityappliance may virtually present a virtual blacked-out object to the userin the shape of the virtual controller.

Some disclosed embodiments may include analyzing the image data todetermine that the second location is edge-free; in response to thedetermination that the second location is edge-free, causing thewearable extended reality appliance to virtually present the virtualcontroller in the second location; analyzing the image data to determinethat the second location includes an edge; and in response to thedetermination that the second area includes an edge, causing thewearable extended reality appliance to perform an action for avoidingpresentation of the virtual controller in the second location. An edgemay include a boundary, brink, corner, end, fringe, line, lip, margin,outskirt, perimeter, rim, side, threshold, tip, verge, bend, bound,brim, contour, curb, frame, periphery, or any other outer limit of thesurface. Adjusting the presentation of the virtual controller based onwhether or not the surface area associated with the second location isedge-free may be desirable in order to ensure that the presentation ofthe virtual controller is not deformed or otherwise flawed because of anobject or background that exists beyond an edge of the surface. Forexample, the image data may be analyzed to determine that a surface areaassociated with the second location is free of corners. If it isdetermined that the surface area associated with the second location isfree of corners, the wearable extended reality appliance may virtuallypresent the virtual controller in the second location. However, if it isdetermined that the surface area associated with the second locationcontains a corner, the wearable extended reality appliance may performan action for avoiding presentation of the virtual controller in thesecond location to avoid contamination of the presented virtual imagewith objects that lie beyond the boundary of the surface, such as shoes,rubbish, or other distracting items.

In some embodiments, the action for avoiding presentation of the virtualcontroller in the second location includes virtually presenting thevirtual controller at a third location proximate the second location.Another surface area in a third location proximate the second locationmay include any area in a location that at least partially avoidspresentation of the virtual controller in the second location. Forexample, when a right corner is detected in the surface area associatedwith the second location, the wearable extended reality appliance mayvirtually present the virtual controller to the left of the detectedcorner.

In some embodiments, the action for avoiding presentation of the virtualcontroller in the second location includes providing a notice via thewearable extended reality appliance, wherein the notice indicates thatthe second location is unsuitable for displaying the virtual controller.The notice may include any or a combination of an alarm, a beep, a textbox, a color change, or any other audio, visual, haptic, or any type ofsensory indication perceivable by a user of the wearable extendedreality appliance. For example, when a corner is detected in the surfacearea associated with the second location, the wearable extended realityappliance may virtually present a virtual blacked-out object to the userin the shape of the virtual controller.

Some disclosed embodiments may include analyzing the image data todetermine that the second location is free of physical objects; inresponse to the determination that the second location is free ofphysical objects, causing the wearable extended reality appliance tovirtually present the virtual controller in the second location;analyzing the image data to determine that the second location includesat least one physical object; and in response to the determination thatthe second location includes at least one physical object, causing thewearable extended reality appliance to perform an action for avoidingcontrol-hampering interference of the physical object with the virtualcontroller. A physical object may include a pencil, paper, phone, or anyother body on the surface that is not an integral element of the surfaceitself. Adjusting the presentation of the virtual controller based onwhether or not the surface area associated with the second location isfree of physical objects may be desirable to ensure that thepresentation of the virtual controller is not deformed or otherwiseflawed because of a distracting object in the background, as well asavoiding a control-hampering interference of the physical object withthe virtual controller. A control-hampering interaction of the physicalobject with the virtual controller may include any communication oraction with the physical object that would impede or obstruct acommunication or action with the virtual controller. For example, theimage data may be analyzed to determine that a surface area associatedwith the second location is free of phones. If it is determined that thesurface area associated with the second location is free of phones, thewearable extended reality appliance may virtually present the virtualcontroller in the second location. However, if it is determined that thesurface area associated with the second location contains a phone, thewearable extended reality appliance may perform an action for avoidingcontrol-hampering interference of the phone with the virtual controllerto avoid causing confusion between interactions of the user with thephone and interactions of the user with the virtual controller.

In some embodiments, the action for avoiding presentation of the virtualcontroller in the second location includes virtually presenting thevirtual controller on a surface of the physical object. This type ofaction may be desirable when the physical object is of a type that wouldnot correspond with a significant control-hampering interaction of thephysical object with the virtual controller. For example, the physicalobject may be a paper that the user would not interact with in the sameway that a user would interact with a virtual keyboard, since the userwould not be typing on the paper. In this example, the action foravoiding control-hampering interference of the paper with the virtualcontroller may be virtually presenting the virtual controller on asurface of the paper. Similar to other embodiments, the action may alsoinclude providing a notice via the wearable extended reality appliance,the notice indicating that the object may affect interactions with thevirtual controller. The notice may include any or a combination of analarm, a beep, a text box, a color change, or any other audio, visual,haptic, or any type of sensory indication perceivable by a user of thewearable extended reality appliance. For example, when a phone isdetected in the surface area associated with the second location, thewearable extended reality appliance may virtually present a virtualblacked-out object to the user in the shape of the virtual controller.

Some disclosed embodiments may include analyzing the image data todetermine a type of the surface at the first location; selecting a firstsize for the virtual controller based on the type of the surface at thefirst location; presenting the virtual controller in the first locationon the surface in the first size; analyzing the image data to determinea type of the surface at the second location; selecting a second sizefor the virtual controller based on the type of the surface at thesecond location; and presenting the virtual controller in the secondlocation on the surface in the second size. A type of the surface mayinclude a color, size, texture, friction coefficient, or any otheraspect of the surface. In one example, adjusting the size of the virtualcontroller based on the type of the surface at a given location may bedesirable to ensure that the virtual controller fits within the space.For example, the surface may be a non-rectangular top of a table, thetop having differing widths at different locations. In this example, theimage data may be analyzed to determine a size of the surface at thefirst location to be 5 feet in width and selecting a 5-foot width forthe virtual controller based on the 5-foot width of the surface at thefirst location. The virtual controller in this example may be presentedin the first location on the surface with a 5-foot width. Thepresentation of the virtual controller in this example may furtherinclude analyzing the image data to determine a size of the surface atthe second location to be 3 feet and selecting a 3-foot width for thevirtual controller based on the 3-foot width of the surface at thesecond location. In this example, the virtual controller may bepresented in the second location on the surface with a 3-foot width. Inanother example, adjusting the size of the virtual controller based onthe type of the surface at a given location may be desirable to ensureadjustment of the size of the virtual controller to a tactile responsewhen interacting with the surface. For example, when a frictioncoefficient of the surface is larger, the size of the virtual controllermay be smaller. In some examples, a visual classification algorithm maybe used to analyze a part of the image data corresponding to a locationon a surface (such as the first location, the second location, etc.) andclassify it to one of a plurality of alternative classes, each class maycorrespond to a type of a surface, and thereby determining the type ofthe surface at the first location and/or at the second location.Further, in response to a first determined type of the surface at thefirst location, one size may be determined as the first size for thevirtual controller, and in response to a second determined type of thesurface at the first location, a different size may be determined as thefirst size for the virtual controller. Further, in response to a thirddetermined type of the surface at the second location, one size may bedetermined as the second size for the virtual controller, and inresponse to a fourth determined type of the surface at the secondlocation, a different size may be determined as the second size for thevirtual controller.

Some disclosed embodiments may include a system for virtually extendinga physical keyboard, the system comprising: at least one processorconfigured to: receive image data from an image sensor associated with awearable extended reality appliance, the image data representing akeyboard placed on a surface; determine that the keyboard is paired withthe wearable extended reality appliance; receive an input for causing adisplay of a virtual controller in conjunction with the keyboard;display, via the wearable extended reality appliance, the virtualcontroller in a first location on the surface, wherein in the firstlocation, the virtual controller has an original spatial orientationrelative to the keyboard; detect a movement of the keyboard to adifferent location on the surface; in response to the detected movementof the keyboard, present the virtual controller in a second location onthe surface, wherein in the second location, a subsequent spatialorientation of the virtual controller relative to the keyboardcorresponds to the original spatial orientation.

Some disclosed embodiments may involve methods for virtually extending aphysical keyboard. FIG. 49 illustrates an exemplary method 4900 forvirtually extending a physical keyboard, consistent with someembodiments of the present disclosure. As shown in step 4910, the method4900 may involve receiving image data from an image sensor associatedwith a wearable extended reality appliance, the image data representinga keyboard placed on a surface. The method 4900 may also involvedetermining that the keyboard is paired with the wearable extendedreality appliance, as shown in step 4912. Step 4914 shows that themethod 4900 may also involve receiving an input for causing a display ofa virtual controller in conjunction with the keyboard. The method 4900may also involve displaying, via the wearable extended realityappliance, the virtual controller in a first location on the surface,wherein in the first location, the virtual controller has an originalspatial orientation relative to the keyboard, as shown in step 4916. Themethod 4900 may further include detecting a movement of the keyboard toa different location on the surface, as shown in step 4918. As shown instep 4920, the method 4900 may also include, in response to the detectedmovement of the keyboard, presenting the virtual controller in a secondlocation on the surface, wherein in the second location, a subsequentspatial orientation of the virtual controller relative to the keyboardcorresponds to the original spatial orientation.

Virtual content may be consumed differently when users are at differentmobility statuses. For example, virtual content may be consumeddifferently when users are stationary (for example, staying at the sameplace, lying down, sitting, standing, etc.) and when users are moving(for example, moving from one place to another, walking, running, etc.).When consuming virtual content in an extended reality environment and/orvia a wearable extended reality appliance, the virtual content mayinterfere with safe and/or efficient movement in a physical environment.For example, the virtual content may hide or obscure elements of thephysical environment, the virtual content may attract the attention ofthe user from elements in the physical environment, and so forth.Therefore, when users are moving, it may be desired to limit thepresentation of the virtual content to minimize the interference of thevirtual content with the movement. On the other hand, when the users arestationary, it may be desired to avoid such limitations to increaseinvolvement, efficiency, and/or immersion of the users. It is thereforedesired to have different presentations of virtual content when theusers are stationary and when the users are moving. Manually switchingbetween the different presentations may be burdensome. Further, usersmight avoid switching and therefore risk injuries while moving. It istherefore desired to switch between the different presentationsautomatically based on the mobility status of the user.

Some disclosed embodiments may include a non-transitory computerreadable medium containing instructions that when executed by at leastone processor cause the at least one processor to perform operations forcoordinate virtual content display with mobility status. Virtual contentmay include a virtual object, inanimate virtual content, animate virtualcontent configured to change over time or in response to triggers,virtual two-dimensional content, virtual three dimensional content, avirtual overlay over a portion of a physical environment or over aphysical object, a virtual addition to a physical environment or to aphysical object, a virtual promotion content, a virtual representationof a physical object, a virtual representation of a physicalenvironment, a virtual document, a virtual character or persona, avirtual computer screen (also referred to as a virtual display herein),a virtual widget, or any other format for displaying informationvirtually. Consistent with the present disclosure, the virtual contentmay include any visual presentation rendered by a computer or aprocessing device. In one embodiment, the virtual content may include avirtual object that is a visual presentation rendered by a computer in aconfined region and configured to represent an object of a particulartype (such as an inanimate virtual object, an animate virtual object,virtual furniture, a virtual decorative object, virtual widget, or othervirtual representation). The rendered visual presentation may change toreflect changes to a status object or changes in the viewing angle ofthe object, for example, in a way that mimics changes in the appearanceof physical objects.

Mobility status may include a position, angle, location, speed,acceleration, activity level, activity type, or any indication of amovement or lack of movement. For example, mobility status may include awalking status, a walking speed, or a walking acceleration. As anotherexample, mobility status may include a sedentary status or a period ofbeing sedentary. Coordinating virtual content display with mobilitystatus may include associating, relating, correlating, coupling,modifying, controlling, or in any other way linking a display of virtualcontent with mobility status. For example, coordinating virtual contentdisplay with mobility status may include displaying certain kinds ofvirtual content when a specific mobility status is present. As anotherexample, coordinating virtual content display with mobility status mayinclude displaying virtual content for a specified period of time when aspecific mobility status is present. In yet another example,coordinating virtual content display with mobility status may includeshowing or hiding certain portions of a virtual content display when aspecific mobility status is present. In an additional example,coordinating virtual content display with mobility status may includechanging a property of the presentation of the virtual content displaybased on the mobility status, such as size, position, opacity,brightness, and so forth.

Some disclosed embodiments may include accessing rules associating aplurality of user mobility statuses with a plurality of display modesfor presenting virtual content via a wearable extended realityappliance. A rule may include a command, direction, formula, guide,model, direction, procedure, specification, or any other principle thatgoverns a state or an action. Accessing a rule may include obtaining,gathering, fetching, referencing, or in any other way obtaining,examining, or retrieving a rule. For example, a rule may be accessedfrom a database by querying for that rule. In some examples, the querymay be triggered by a condition being met. In other examples, the querymay be triggered by a user. In one example, accessing one or more rulesmay include accessing computer instructions defining an algorithmcorresponding to the one or more rules. In another example, accessingone or more rules may include accessing a data-structure associatingmobility statuses with display modes that is configured to enableretrieval of a display mode based on a mobility status (such as a mapdata-structure, a dictionary data-structure, and so forth). A usermobility status may include a position, angle, location, speed,acceleration, activity level, activity type, or any indication of amovement or lack of movement of a user, as opposed to any suchindication of another object. For example, a user mobility status mayinclude a walking status, a walking speed, or a walking acceleration ofa user. As another example, mobility status may include a sedentarystatus of a user or a period of a user being sedentary. A display modemay include a color, brightness, opacity, angle, perspective,visibility, region of display, size of display, indication of types ofobjects shown, number of objects shown, or any other visual aspect of adisplay. For example, a display mode may include a document beingpresented with no information obscured. In another example, a displaymode may include a document being presented with some portions obscuredby blurring and other portions unobscured. In yet another example, afirst display mode may include a presentation of virtual content (suchas a particular virtual object) in a first size and at a first region,and a second display mode may include a presentation of the same virtualcontent (for example, the particular virtual object) in a second size(that may be different from the first size) and at a second region (thatmay be different from the first region).

In some embodiments, the accessed rules may associate display modes withuser mobility statuses that include at least two of a sitting status, astanding status, a walking status, a running status, a cycling status,or a driving status. A sitting status may include a status of a usersitting at a chair, sitting in a car, sitting on the floor, or any otherstatus associated with the user being seated on any surface. A standingstatus may include a status of a user standing up straight, reaching up,stretching, or any other status associated with the user having ormaintaining an upright or nearly upright position, supported by theuser's feet. A walking status may include a status of a user hiking,jaunting, parading, stepping, strolling, sauntering, striding, or anyother status associated with the user moving along on foot. A runningstatus may include a status of a user racing, rushing, spurting,darting, dashing, jogging, sprinting, or any other status associatedwith the user moving along on foot at a speed that is faster thanwalking. A cycling status may include a status of a user riding a cycle,sitting on a cycle, riding an exercise cycle, or sitting on an exercisecycle. It is to be understood that a cycle may include any number ofwheels. For example, a cycle may include a uni-cycle, a bicycle, atricycle, or any other type of cycle. A driving status may include astatus of a user driving a vehicle, sitting in a vehicle, starting avehicle, or any other status associated with the user interacting withany type of motor vehicle.

In some embodiments, the accessed rules may associate user mobilitystatuses with the plurality of display modes that include at least twoof a working mode, an entertainment mode, a sport activity mode, anactive mode, a sleep mode, a tracking mode, a stationary mode, a privatemode, or a public mode. A working mode may be a display mode including adisplay involving documents, spreadsheets, messages, assignments, or anyother information that may be relevant to any activity involving mentalor physical effort done to achieve a purpose or result. An entertainmentmode may be a display mode including a display involving television,movies, images, games, or any other information that may be relevant toany activity involving either providing or being provided with amusementor enjoyment. A sport activity mode may be a display mode including adisplay involving a game, athletic data, team statistics, playerstatistics, or any other information that may be relevant to anyactivity including physical exertion and skill in which an individual orteam competes against another or others. An active mode may be a displaymode including a display involving fitness statistics, heart rate,metabolism, speed, personal records, height, weight, or any otherinformation that may be relevant to any activity requiring physicaleffort that may be carried out to sustain or improve health and fitness.A sleep mode may be a display mode including a display involving sleepduration, sleep cycles, sleep conditions, sleep disorders, or any otherinformation that may be relevant to any activity involving the naturalstate of rest during which an individual's eyes are closed and theindividual becomes unconscious. A tracking mode may be a display modeincluding a display involving a fitness tracker, a heart rate meter, aweather tracker, or any other information that may be relevant tofollowing the course of any type of information. A stationary mode maybe a display mode including a display involving a television program, amovie, documents, or any other type of information that may beappropriate or desirable for a user in a seated or otherwise sedentaryposition. A private mode may be a display mode including a displayinvolving blurred, blacked out, or otherwise obscured portions, or anyother type of information that is presented in a way to limit its accessor use to one particular person or group of people. A public mode may bea display mode including a display involving unobscured portions, widelyavailable information, websites, or any other type of information thatpertains to or is able to be accessed by anyone.

In some embodiments, each of the plurality of display modes may beassociated with a specific combination of values of a plurality ofdisplay parameters, and the operations may further include receivinginput from the user to adjust values of display parameters associatedwith at least one display mode. A display parameter may include anyvalue or data associated with a visual presentation, In someembodiments, the plurality of display parameters may include at leastsome of an opacity level, a brightness level, a color scheme, a size, anorientation, a resolution, displayed functionalities, or a dockingbehavior. For example, a private display mode may be associated with aspecific combination of values for an opacity level and a brightnesslevel, so that in a private display mode, the displayed information maynot be highly visible. Receiving input from the user may include anyuser interaction with any physical device configured to receive inputfrom a user or an environment of the user, and to provide the data to acomputational device. The data provided to the computational device maybe in a digital format and/or in an analog format. In one embodiment,the input device may store the input received from the user in a memorydevice accessible by a processing device, and the processing device mayaccess the stored data for analysis. In another embodiment, the inputdevice may provide the data directly to a processing device, forexample, over a bus or over another communication system configured totransfer data from the input device to the processing device. In someexamples, the input received by the input device may include keypresses, tactile input data, motion data, position data, direction data,or any other data for supply for computation. Some examples of the inputdevice may include a button, a key, a keyboard, a computer mouse, atouchpad, a touchscreen, a joystick, or another mechanism from whichinput may be received. Adjusting values of the display parameters mayinclude increasing, decreasing, replacing, removing, or in any other waymodifying the values. For example, receiving input from the user toadjust values of display parameters may include the user typing into akeyboard a value of five to replace a zoom value of four associated withthe display, wherein the modification would result in a more zoomed-indisplay.

Some disclosed embodiments may include receiving first sensor data fromat least one sensor associated with the wearable extended realityappliance, the first sensor data may be reflective of a mobility statusof a user of the wearable extended reality appliance during a first timeperiod. Sensor data reflective of a mobility status of the user mayinclude a position, location, angle, speed, acceleration, elevation,heart rate, or any other information obtainable by any device thatdetects or measures a physical property associated with a movement or alack of movement. For example, sensor data reflective of a mobilitystatus may include a speed measurement that reflects a user moving atthat measured speed. In another example, sensor data reflective of amobility status may include a measurement of a change in location thatreflects a user moving from one location to another. As another example,sensor data reflective of a mobility status may include a heart ratemeasurement indicating that a user is exercising, or a heart ratemeasurement indicating that a user is sedentary. A time period mayinclude a cycle, date, duration, span, stretch, term, interval, or anyextent of time. For example, sensor data reflective of a mobility statusof a user of the wearable extended reality appliance during a first timeperiod may include an average measurement of speed over thirty secondsthat indicates whether the user moved by a significant amount, orwhether the user merely had a brief spurt of movement.

Some disclosed embodiments may include, based on the first sensor data,determining that during the first time period the user of the wearableextended reality appliance is associated with a first mobility status.Such an association may be accomplished through any means of linkinginformation, such as referencing a data structure, database, or a lookuptable that correlates sensor data with mobility status. For example, thefirst sensor data may include an average measurement of speed of threemiles per hour over ten minutes, which may be associated with a walkingmobility status. In another example, the first sensor data may include aheart rate measurement of 150 beats per minute, which may be associatedwith an exercise mobility status. In some examples, a machine learningmodel may be trained using training examples to determine mobilitystatues associated with users of wearable extended reality appliancesfrom sensor data. An example of such training example may include asample sensor data captured using a sample wearable extended realityappliance, together with a label indicating a mobility status associatedwith a user of the sample wearable extended reality appliance in a timeperiod corresponding to capturing of the sample sensor data. In oneexample, the trained machine learning model may be used to analyze thefirst sensor data and determine that during the first time period theuser of the wearable extended reality appliance is associated with thefirst mobility status. In another example, the trained machine learningmodel may be used to analyze the second sensor data (described below)and determine that during the second time period (described below) theuser of the wearable extended reality appliance is associated with thesecond mobility status (described below).

Some disclosed embodiments may include implementing at least a firstaccessed rule to generate a first display of the virtual content via thewearable extended reality appliance associated with the first mobilitystatus. A display of the virtual content may include a full or partialpresentation of the virtual content. For example, a first display of thevirtual content may include a full, unobscured document. In anotherexample, a first extended reality display of the virtual content mayinclude a document with certain portions obscured by blurring and otherportions unobscured. A display of the virtual content may also includeany portion or all of a virtual object, virtual screen (also referred toas virtual display herein), or virtual scene presented in any color,shape, dimension, perspective, or any other type of visible attribute ofan object. For example, a display of the virtual content may include avirtual tree in its original size. In another example, a display of thevirtual content may include a zoomed-in virtual tree. Implementing anaccessed rule to generate a display of the virtual content via thewearable extended reality appliance associated with the first mobilitystatus may include referencing the rule in any way, such as via a lookuptable or as described above, that associates a display mode with thefirst mobility status, and presenting a presentation of the virtualcontent that is based on the display mode via the wearable extendedreality appliance. For example, when the first mobility status isdetermined to be walking, the processor may reference a lookup table toidentify a rule that correlates a walking status with a display mode ofa virtual content away from a direction of movement of the user. In suchan embodiment, the processor may display the virtual content via thewearable extended reality appliance in a region that is not in thedirection of movement of the user.

Some disclosed embodiments may include receiving second sensor data fromthe at least one sensor, the second sensor data being reflective of themobility status of the user during a second time period. The secondsensor data reflective of a mobility status of the user may be receivedin a manner similar to that discussed above for the first sensor data.The second sensor data may be of a similar type to the first sensordata. For example, both the first sensor data and the second sensor datamay include a speed measurement. The second sensor data may also be of adifferent type than the first sensor data. For example, the first sensordata may include a speed measurement, while the second sensor data mayinclude a heart rate measurement. The second time period may havecharacteristics similar to the first time period discussed above. Forexample, sensor data reflective of a mobility status of a user of thewearable extended reality appliance during a second time period mayinclude an average measurement of speed over thirty seconds thatindicates whether the user moved by a significant amount, or whether theuser merely had a brief spurt of movement. The second time period may besimilar in duration to the first time period. For example, both thefirst time period and the second time period may be ten minutes. Thesecond time period may also include a different duration than the firsttime period. For example, the first time period may be five minutes,while the second time period may be fifteen minutes.

Some disclosed embodiments may include based on the second sensor data,determining that during the second time period the user of the wearableextended reality appliance is associated with a second mobility status.Such an association may be accomplished through similar linking means asdiscussed above for the first sensor data. The second mobility statusmay be similar to the first mobility status. For example, the firstmobility status and the second mobility status may both be a walkingstatus. Alternatively, the second mobility status may be different fromthe first mobility status. For example, the first mobility status may bea sitting status, while the second mobility status may be a standingstatus.

Some disclosed embodiments may include implementing at least a secondaccessed rule to generate a second display of the virtual content viathe wearable extended reality appliance associated with the secondmobility status, wherein the second display of the virtual content maydiffer from the first display of the virtual content. A second displayof the virtual content may be accomplished in a manner similar to thatdiscussed above for the first display of the virtual content.Implementing an accessed rule to generate a display of the virtualcontent via the wearable extended reality appliance associated with thesecond mobility status may include referencing the rule in any way, suchas a lookup table or as described above, that associates a display modewith the second mobility status, and presenting a presentation of thevirtual content that is based on the display mode via the wearableextended reality appliance. For example, when the second mobility statusis determined to be sitting, the processor may reference a lookup tableto identify a rule that correlates a sitting status with a display modeof a virtual content in front of the user. In such an embodiment, theprocessor may display the virtual content in a region that is in frontof the user. A second display of the virtual content may differ from thefirst display of the virtual content in terms of portions shown, color,dimension, perspective, angle, region of presentation, opacity, size, orany other visual aspect of the virtual content. For example, a firstdisplay of the virtual content may include a color image, while a seconddisplay of the virtual content may include a black and white image. Inanother example, a first display of the virtual content may include amoving scene, while a second display of the virtual content may includea still scene. In yet another example, a first display of the virtualcontent may include presenting the virtual content away from a directionof motion of the user, while a second display of the virtual content mayinclude presenting the virtual content in all directions. In anadditional example, a first display of the virtual content may includepresenting the virtual content with one opacity, while a second displayof the virtual content may include presenting the virtual content with adifferent opacity. In an additional example, a first display of thevirtual content may include presenting the virtual content in a firstsize, while a second display of the virtual content may includepresenting the virtual content in a different size.

In some examples, the second time period may be later than the firsttime period. Further, implementing the at least second accessed rule togenerate the second display of the virtual content via the wearableextended reality appliance may comprise causing the display of thevirtual content to gradually transition from the first displayassociated with the first mobility status to the second displayassociated with the second mobility status. For example, the virtualcontent may comprise a plurality of elements, and the graduallytransition may comprise transitioning specific elements from the firstdisplay associated with the first mobility status to the second displayassociated with the second mobility status when the user is notinteracting with the specific elements, and withholding transitioningspecific elements from the first display associated with the firstmobility status to the second display associated with the secondmobility status when the user is interacting with the specific elements.In another example, the virtual content may comprise a plurality ofelements, and the gradually transition may comprise transitioningspecific elements from the first display associated with the firstmobility status to the second display associated with the secondmobility status when the specific elements are outside a field of view(for example, of the user, of the wearable extended reality appliance,etc.), and withholding transitioning specific elements from the firstdisplay associated with the first mobility status to the second displayassociated with the second mobility status when the specific elementsare in the field of view.

FIGS. 50A to 50D illustrate examples of various virtual content displayscoordinated with different mobility statuses, consistent with someembodiments of the present disclosure. As illustrated in FIG. 50A, whena user is in a sitting mobility status 5010, a first display 5012 of thevirtual content may be presented. For example, when a user is sitting,the wearable extended reality appliance may present virtual content thatmay be more appropriate to view or interact with in a sitting position,such as a document for viewing or editing. As illustrated in FIG. 50B,when a user is in a standing mobility status 5014, a second display 5016of the virtual content may be presented. For example, when a user isstanding, the wearable extended reality appliance may present virtualcontent that may be more appropriate to view or interact with in astanding position, such as a notification of a message. As illustratedin FIG. 50C, when a user is in a walking mobility status 5018, a thirddisplay 5020 of the virtual content may be presented. For example, whena user is walking, the wearable extended reality appliance may presentvirtual content that may be more appropriate to view or interact with ina walking position, such as a step counter. As illustrated in FIG. 50D,when a user is in a running mobility status 5022, a fourth display 5024of the virtual content may be presented. For example, when a user isrunning, the wearable extended reality appliance may present virtualcontent that may be more appropriate to view or interact with in arunning position, such as heart rate dynamics of the user while the useris running.

In some embodiments, when a user is in a sitting mobility status 5010, afirst display of the virtual content may be presented. For example, whena user is sitting, the wearable extended reality appliance may presentthe virtual content (such as a virtual document) in an orientation or alocation appropriate for viewing or interaction in a sitting position,such as in front of the user, for example at or near an eye level of theuser. Further, when a user is in a standing mobility status 5014, asecond display of the virtual content may be presented. For example,when a user is standing, the wearable extended reality appliance maypresent the virtual content (such as the virtual document) in anorientation or a location appropriate to viewing or interaction in astanding position, such as in front of the user, for example at or belowan eye level of the user. Further, when a user is in a walking mobilitystatus 5018 or in a running mobility status 5022, a third display of thevirtual content may be presented. For example, when a user is walking,the wearable extended reality appliance may present the virtual content(such as the virtual content) in an orientation or a locationappropriate to viewing or interaction in a walking position, such asaway from a moving direction of the user, and/or significantly below aneye level of the user.

In some embodiments, when a user is in a sitting mobility status 5010, afirst display of the virtual content may be presented. For example, whena user is sitting, the wearable extended reality appliance may presentthe virtual content (such as a virtual display) in a size and/or with anopacity appropriate for viewing or interaction in a sitting position,for example in a large size and/or with high opacity. Further, when auser is in a walking mobility status 5018 or in a running mobilitystatus 5022, a second display of the virtual content may be presented.For example, when a user is walking or running, the wearable extendedreality appliance may present the virtual content (such as the virtualdisplay) in a size and/or with an opacity appropriate to viewing orinteraction in a walking or running position, such as in a smaller sizeand/or with a lower opacity than when the user is sitting.

Some disclosed embodiments may include determining the mobility statusof the user during the first time period based on the first sensor dataand history data associated with the user, and determining the mobilitystatus of the user during the second time period based on the secondsensor data and the history data. History data may include data that waspreviously acquired by the processor either automatically or throughuser input. Data that was previously acquired by the processorautomatically may include data that was saved by the processor becauseit met a certain condition sufficient for acquisition. For example, theprocessor may be configured to save as history data, any sensor datathat is similar to sensor data indicative of a certain mobility status,to better train the processor, for example via a machine learningalgorithm, to identify sensor data that may indicate that mobilitystatus. Data that was previously acquired by the processor through userinput may include data that was entered into the processor by the userinteracting with a mouse, touch pad, keyboard, or any other devicecapable of converting user interaction into information that may besaved by the processor. For example, a user may associate certain sensordata with an exercise status that the processor does not automaticallyassociate with an exercise status. In such an example, the user mayenter that sensor data to be associated with an exercise status ashistory data, so that in the future, the processor may associate thatsensor data with an exercise status. The use of history data in this waymay be desirable due to mobility differences between different users.For example, for some users, a certain speed may be associated with arunning mobility status, while for another user, the same speed may beassociated with a walking mobility status. The history data may includeany information regarding or measured from a user, a sensor, a wearableextended reality device, a device connected to or used with a wearableextended reality device, or any information that may be input into theprocessor. For example, the history data may include a useridentification, such as the name, weight, or height of the user. Inother examples, the history data may include details of the wearableextended reality appliance being used by the user, such as a devicetype, a device model number, or a device configuration.

In some embodiments, the at least one sensor may include an image sensorwithin the wearable extended reality appliance, and the operations mayfurther include analyzing image data captured using the image sensor toidentify a switch between the first mobility status and the secondmobility status. An image sensor may include a charge coupled device(CCD), complementary metal oxide semiconductor (CMOS), or any otherdevice that detects and conveys information related to an image. Imagedata may include pixel information, image size, resolution, depth offield, type of objects, number of objects, and any other informationthat may be related to an image. A switch between the first mobilitystatus and the second mobility status may include a change in speed,position, posture, activity, location, or any other transition betweenthe first mobility status and the second mobility status. For example, aCCD device in a camera incorporated into the wearable extended realityappliance may detect a user in a first mobility status where the user isstanding, and determine that the mobility status has switched to asecond mobility status of walking based on a change in objects that theCCD device detects in an image. Additionally, or alternatively, forexample, the CCD device may determine a change in the mobility statusbased on a change in a speed or an acceleration of one or more objectsdetected in the image. In one example, the image data may be classifiedusing a binary visual classification algorithm to one of two alternativeclasses, ‘switching between mobility statuses’ and ‘no switch betweenmobility statuses’, and thereby identify the switch in the mobilitystatus. In another example, the image data may be classified using amulticlass classifier to one of three or more alternative classes, oneof the alternative classes may correspond to no switch in mobilitystatus, and any other class of the alternative classes may correspond toa switch from one particular class to another particular class (forexample, ‘from sitting to walking’, ‘from standing to sitting’, ‘fromwalking to running’, and so forth). In yet another example, the imagedata may be analyzed using an ego-motion algorithm to measure an amountof movement of the at least one image sensor within the wearableextended reality appliance (and therefore an amount of movement of thewearable extended reality appliance), and the identification of theswitch between the first mobility status and the second mobility may bebased on the amount of motion (for example, based on a result of acomparison of the amount of motion with a selected threshold, or asdescribed below).

In some embodiments, the at least one sensor may include at least onemotion sensor included in a computing device connectable to the wearableextended reality appliance, and the operations may further includeanalyzing motion data captured using the at least one motion sensor toidentify a switch between the first mobility status and the secondmobility status. A motion sensor may include a passive infrared sensor,a microwave sensor, an area reflective sensor, an ultrasonic sensor, avibration sensor, an accelerometer, or any other type of device that maybe used to measure a movement of an object or surface. In anotherexample, a motion sensor may include an image sensor, and image datacaptured using the image sensor may be analyzed using an ego-motionalgorithm and/or a visual localization algorithm to determine themotion. Motion data may include position, distance, dimension, angle,speed, acceleration, rotation, or any other information regarding amovement of an object or surface. A switch between the first mobilitystatus and the second mobility status may include a change in speed,position, posture, activity, location, or any other transition betweenthe first mobility status and the second mobility status. For example,an infrared sensor incorporated into a housing of the wearable extendedreality appliance may detect a user in a first mobility status where theuser is standing, and determine that the mobility status has switched toa second mobility status of walking based on a change in speed detectedby the infrared sensor. The motion sensor may be connectable to thewearable extended reality appliance in a plurality of ways. In certainembodiments, the motion sensor may be incorporated within the wearableextended reality appliance. In other embodiments, the motion sensor maybe connected to the wearable extended reality appliance by a wiredconnection. In yet other embodiments, the motion sensor may be connectedto the wearable extended reality appliance by a wireless connection. Awireless connection may utilize WiFi, Bluetooth™, or any other channelthat transmits information without a wired connection between the motionsensor and the wearable extended reality appliance. For example, thewearable extended reality appliance may be a smartphone that iswirelessly paired via Bluetooth™ to an ultrasonic sensor.

Some disclosed embodiments may include during the first time perioddisplaying a certain virtual object in a working mode and during thesecond time period displaying the certain virtual object in a sportactivity mode. A virtual object may include a visual presentationrendered by a computer and configured to represent an object of aparticular type, such as an inanimate virtual object, an animate virtualobject, virtual furniture, a virtual decorative object, virtual widget,or other virtual representation. For example, a virtual object mayinclude a virtual window, and the extended reality display may include afirst virtual window including a document and a second virtual windowincluding statistics of a sports team for which the user plays. In aworking mode, the first virtual window including the document may bepresented on top of the second virtual window including statistics ofthe sports team. In a sport activity mode, the first virtual windowincluding the document may be underneath the second virtual windowincluding statistics of the sports team. In another example, in aworking mode the virtual object may be located at a fixed location inthe extended reality environment and may not move from that location dueto small movements of the wearable extended reality appliance, while inthe sport activity mode the virtual object may be configured to move inthe extended reality environment with the movements of the wearableextended reality appliance.

Some disclosed embodiments may include during the first time perioddisplaying a certain virtual object in an active mode and during thesecond time period displaying the certain virtual object in a sleepmode. Displaying the virtual object in an active or sleep mode duringdifferent time periods may be desirable to present a display that isbetter suited for the given time period. In some time periods, a usermay be more likely to be in an active mode, such as during the day, whenthe user is not in a lying position, and so forth. During such times, itmay be desirable to present the virtual object in a mode that isconsistent with being active during the day. In other time periods, auser may be more likely to be in a sleep mode, such as during the night,when the user is in a lying position, and so forth. During such times,it may be desirable to present the virtual object in a mode that isconsistent with sleeping during the night. For example, the virtualobject may include a virtual clock. In an active mode, the virtual clockmay be displayed with a maximum brightness so that the user can see theclock well while moving around. In a sleep mode, the virtual clock maybe displayed with a minimum brightness, so that the clock does notdisrupt the user's sleep.

Some disclosed embodiments may include during the first time perioddisplaying a certain virtual object in a private mode and during thesecond time period displaying the certain virtual object in a publicmode. Displaying the virtual object in a private or public mode duringdifferent time periods may be desirable to preserve privacy of theinformation presented through the virtual object during times where itis appropriate. In some time periods, a user may be more likely to be ina private setting, such as home. During such times, it may be desirableto present the virtual object in a mode that is consistent with viewingat home. In other time periods, a user may be more likely to be in apublic setting, such as the mall or a park. During such times, it may bedesirable to present the virtual object in a mode that is consistentwith viewing among individuals that do not have access to privateinformation contained in the virtual object. For example, the virtualobject may include a virtual document containing portions intended foraccess only by the user and portions intended for access by anyone. In aprivate mode, the virtual document may be presented with the entiredocument unobscured. In a public mode, the virtual document may bepresented with the portions intended for access only by the userobscured by blurring, blacking out, or by using any other manner ofpreventing its access by anyone other than the user.

In some embodiments, generating the first display associated with thefirst mobility status includes using a first display mode for displayinga first virtual object and using a second display mode for displaying asecond virtual object. Associating a display mode with each virtualobject in the display may be desirable to present virtual objects withcharacteristics that are appropriate for the type of the virtual object.Some virtual objects may be better suited for a private display mode,such as a document containing sensitive, private information. Othervirtual objects may be better suited for a public display mode,including information that a user would like to share with someone elsefreely, such as the time. It may be undesirable to associate a singledisplay mode with both of these virtual objects because either theprivate document would become publicized, or the user would not be ableto share the time information with someone else. Therefore, it may bedesirable to associate a display mode with each virtual object in thedisplay. The first display mode and the second display mode may besimilar. For example, both the first display mode and the second displaymode may be in color. In another example, both the first display modeand the second display mode may be unobscured. Alternatively, the firstdisplay mode and the second display mode may be of differing types. Forexample, the first virtual object may be a clock and the second virtualobject may be a photo. In this example, the clock may be displayed in afirst display mode in color and the photo may be displayed in a seconddisplay mode in black and white. In another example, the first virtualobject may be a calendar and the second virtual document may be aprivate document. In this example, the calendar may be displayed in afirst display mode that is unobscured, and the private document may bedisplayed in a second display mode that is blurred out.

Some disclosed embodiments may include during the first time perioddisplaying the first virtual object in a public mode and the secondvirtual object in a private mode. For example, the first virtual objectmay be a virtual document and the second virtual object may be a virtualball. This type of display may be desirable during a time period wherethe user may need to access both private and public information, whilethe user is in a public location, such as a mall or a park. By providingfor one virtual object to be displayed in a public mode and another tobe displayed in a private mode during such a time period, the user maybe able to view all of the objects they want to view without worryingabout exposing any private information to others. In a public mode, thevirtual document may be presented with portions that are not intendedfor general access being blurred or blacked out. In a private mode, thevirtual ball may be displayed unobscured to the user. In anotherexample, the first virtual object may be a virtual window of a shoppingapplication and the second virtual object may be a virtual window of abanking application. In a public mode, the virtual window of theshopping application may be displayed with all of the items for salepresented unobscured, while any portions associated with a paymentinformation being presented obscured by blurring or blacking out theportions. In a private mode, the virtual window of the bankingapplication may be displayed with all of its portions unobscured.

Some disclosed embodiments may include during the second time periodchanging the first display mode of the first virtual object andmaintaining the second display mode of the second virtual object.Changing one display mode while maintaining the other may be desirableto associate a more appropriate display mode for a virtual object duringthe second time period. In some time periods, a user may be more likelyto have left from one location where one display mode is appropriate toanother location where another display mode is more appropriate. Duringsuch times, it may be desirable to only change one display mode, whilemaintaining the other, so that the user does not have to view bothvirtual objects in the same display mode if that one display mode is notappropriate at the other location. For example, the first virtual objectmay be a virtual clock and the second virtual object may be a virtualcalendar. During the first time period, such as when the user is at asporting event, the virtual clock may be displayed in color and thevirtual calendar may be displayed in black and white. During such atime, it may be appropriate to have a contrast between the two virtualobjects and emphasize the clock, so that the user can better payattention to time during the sporting event. In this example, during thesecond time period, such as when the sporting event is over or the userhas returned home, the virtual clock may be displayed in black andwhite, as changed from the color display mode, while the virtualcalendar may be maintained in a black and white display mode. Duringsuch a time, it may be appropriate to not have a contrast between thetwo virtual objects and to not emphasize the clock, since the user maynot need to actively look at the time.

Some disclosed embodiments may include during the second time periodusing a third display mode for displaying the first virtual object andthe second virtual object. This type of display may be desirable topresent all virtual objects in a single display mode during a time whensuch a display is appropriate. In some time periods, the user may notwant or need the various virtual objects to be presented in differentdisplay modes. Some situations where this may apply may include wherethe user would like the virtual objects to be in a background viewrelative to a physical object the user is viewing, or where the user maywant to look at all of the virtual objects in the same way. The thirddisplay mode may be the same as the first display mode and the seconddisplay mode. Alternatively, the third display mode may be differentfrom the first display mode and/or the second display mode. For example,the first virtual object may be a virtual clock and the second virtualobject may be a virtual calendar. During the first time period, thevirtual clock may be displayed in color and the virtual calendar may bedisplayed in black and white. During the second time period, the virtualclock may be displayed in black and white, as changed from the colordisplay mode, while the virtual calendar may be maintained in a blackand white display mode. In this example, during the second time period,the virtual clock and the virtual calendar may be displayed at a 50%opacity level.

In some embodiments, the accessed rules may further associate differentdisplay modes with different types of virtual objects for differingmobility statuses. As discussed above, one or more rules may associatedisplay modes with mobility statuses. In some embodiments, the one ormore rules may correlate different display modes with different types ofvirtual objects. A virtual object may include any visual presentationrendered by a computer in a confined region and configured to representan object of a particular type, as described above. Different displaymodes may be desirable for different types of virtual objects to presentthe virtual objects with characteristics that may be more appropriatefor that type of virtual object. Some virtual objects, such as documentsor webpages, may require display modes associated with blurring orflagging certain portions of the text for privacy or highlightingapplications. Other virtual objects, such as calendars or maps, mayrequire display modes associated with different coloring schemes,dimensions, or sizes associated with displaying dates, events, ordirections. It may not be desirable or efficient to have all displaymodes associated with all virtual objects in such situations becauseblurring may not apply for displaying certain calendar events, whiledifferent coloring schemes may not apply for displaying certaindocuments. For example, the accessed rules may associate display modesinvolving text size for virtual objects that include text, such asdocuments and spreadsheets, while the accessed rules may associatedisplay modes involving blurring or blacking out for virtual objectsthat include sensitive information, such as a virtual bankingapplication window containing a user's personal financial information.

In some embodiments, the different types of virtual objects may includeat least two of: work-related virtual objects, health-related virtualobjects, travel-related virtual objects, finance-related virtualobjects, sport-related virtual objects, social-related virtual objects,docked virtual objects, undocked virtual objects. It may be desirable tohave more than one type of virtual object so that the user of thewearable extended reality appliance may view multiple types ofinformation in a single display, instead of having to shift to adifferent display to view each type of information. This may allow for agreater amount of information conveyed, as well as enablingmulti-tasking among many different tasks associated with different typesof virtual objects. Work-related virtual objects may include one or moreof a planner, schedule, document, spreadsheet, or any other objectassociated with any activity involving mental or physical effort done toachieve a purpose or result. Health-related virtual objects may includeone or more of a fitness tracker, a heart rate monitor, a weightmonitor, a height monitor, or any other object associated with anyphysiological aspect. Travel-related virtual objects may include one ormore of a map, ticket, traffic monitor, pass, or any other objectassociated with any movement from one place to another. Finance-relatedvirtual objects may include one or more of a stock ticker, bankingapplication, currency converter, or any other object associated with anymanagement of money. Sport-related virtual objects may include one ormore of a speed meter, a pacer, exercise tracker, or any other objectassociated with any activity involving physical exertion and skill inwhich an individual or team competes against another or others forentertainment. Social-related virtual objects may include one or more ofa social media application, a messenger, a text message, a notification,or any other object associated with any creation or sharing of, orparticipation in networking. Docked virtual objects may include anyobject that is connected, tethered, linked, or otherwise bound to anarea or another object. For example, a virtual keyboard may be docked toa physical keyboard, such that the virtual keyboard moves in alignmentwith the physical keyboard. Undocked virtual objects may include anyobject that is not connected, tethered, linked, or otherwise bound to anarea or another object. For example, a virtual keyboard that is undockedfrom a physical keyboard may not move in alignment with the physicalkeyboard, and remain static, or move independently as the physicalkeyboard moves. In some situations, it may be desirable to present atleast two of the different types of virtual objects. For example, when auser is working remotely while traveling, it may be desirable to displayboth travel-related virtual objects and work-related virtual objects, sothat the user can track their travel on a map, while also working on aspreadsheet for work.

In some embodiments, each type of virtual object may be associated witha priority, and the association of different display modes withdifferent types of virtual objects for differing mobility statuses maybe based on the priorities associated with the different types ofvirtual objects. Associating a type of virtual object with a prioritymay be desirable to display certain virtual objects with more urgency,highlight, or any other type of importance as compared to other virtualobjects. Some display modes may be associated with higher importance,such as increased brightness modes, while other display modes may beassociated with lower importance, such as decreased brightness modes. Apriority may include an arrangement, preference, order, rank, seniority,superiority, numbering, lettering, or any other condition of beingregarded or treated as more important. In some embodiments, the prioritymay be provided by user input, similar to the user input describedabove. In other embodiments, the processor may automatically associate apriority with each type of virtual object based on predefinedparameters, database values, or any other means of electronicallyassociating one value with another. For example, a notification type ofvirtual object may be associated with a higher priority than a videotype of virtual object. In this example, the higher-ranked notificationtype may be associated with display modes that favor presentation on topof other virtual objects, such as increased brightness, opacity, orcontrast, or certain color schemes, such as a yellow tone, while thelower-ranked video type may be associated with display modes that favorpresentation underneath other virtual objects, such as reducedbrightness, opacity, or contrast, or certain other color schemes, suchas black and white.

Some disclosed embodiments may include presenting via the wearableextended reality appliance a first virtual object associated with afirst type and a second virtual object associated with a second type,wherein generating the first display associated with the first mobilitystatus may include applying a single display mode for the first andsecond virtual objects, and generating the second display associatedwith the second mobility status may include applying differing displaymodes for the first and second virtual objects. This type of display maybe desirable in situations where a user is going from one type ofactivity requiring uniform display modes, such as sitting at a desk inthe user's home, to another type of activity requiring different displaymodes, such as walking around town. While at home, it may be desirableto present both virtual objects in a public mode where both objects areunobscured because, in the privacy of the user's home, the user may notneed to be concerned about others accessing private information in oneof the virtual objects. However, when the user is walking around town,it may be desirable to present the virtual objects in different displaymodes, where one object may be obscured and the other may be unobscured,to protect the user's information while in a public setting, while stillallowing the user to view the desired information in both of the virtualobjects. FIGS. 51A and 51B illustrate examples of different displaymodes associated with different types of virtual objects for differingmobility statuses, consistent with some embodiments of the presentdisclosure. In FIG. 51A, the first virtual object is a calendarapplication 5112 and the second virtual object is a banking application5114. When the mobility status is stationary 5110, both the calendarapplication 5112 and the banking application 5114 may be displayed in aworking display mode 5116, which may be an unobscured display of anapplication. In FIG. 51B, the same calendar application 5112 and bankingapplication 5114 are displayed. When the mobility status is walking5118, the calendar application 5112 may be displayed in the workingdisplay mode 5116, while the banking application 5114 may be displayedin a private display mode 5120, which may be an obscured display of anapplication, such as a blurred display.

In some embodiments, the accessed rules may further associate theplurality of user mobility statuses with a plurality of display modesbased on an environmental context. An environmental context may includea climate, habitat, setting, situation, status, surrounding, ambiance,background, circumstance, context, locale, neighborhood, scenery,terrain, territory, land, structure, or any other fact or conditionassociated with a location of a user. Associating the user mobilitystatuses with display modes based on an environmental context may bedesirable to make the display more efficient. By associating the displaymodes based on the environmental context, only the display modes thatare appropriate for that environmental context may be used, instead ofall of the possible display modes, which may improve processorefficiency and speed. This may be desirable in situations where certaindisplay modes would rarely, if at all, be appropriate in a givenenvironmental context, such as private display modes in very publiclocations, such as the mall. FIGS. 52A and 52B illustrate examples ofdifferent display modes associated with differing mobility statusesbased on an environmental context, consistent with some embodiments ofthe present disclosure. In FIG. 52A, when a walking mobility status 5210is determined, an environmental context of a park location 5212 maycause the walking mobility status 5210 to be associated with a firstdisplay mode 5214 in the form of a step counter, to help the user whomay be on a hike in the park. In FIG. 52B, an environmental context of acity location 5216 may cause the walking mobility status 5210 to beassociated with a second display mode 5218 in the form of a map, to helpthe user navigate around the city. In another example, when a walkingmobility status is determined, an environmental context of nearbyobstacles may cause the walking mobility status to be associated with afirst display mode in the form of removing all virtual objects from thedirection of movement of the user (for example, remove completely frompresentation, remove to the side, etc.), while an environmental contextof no nearby obstacles may cause the walking mobility status to beassociated with a second display mode in the form of showing virtualobjects in the direction of movement of the user with low opacity. Inyet another example, when a siting status is determined, anenvironmental context of nearby people may cause the sitting mobilitystatus to be associated with a first display mode in the form of partialimmersion, while an environmental context of no nearby people may causethe sitting mobility status to be associated with a second display modein the form of full immersion.

In some embodiments, the environmental context is based on at least oneof a location of the wearable extended reality appliance, an orientationof the wearable extended reality appliance, or a current time. It may bedesirable to have the environmental context be based on location,orientation, or time because these factors are indications of asurrounding of a user. A location of the wearable extended realityappliance may provide information regarding whether a user is in anoutdoor, indoor, private, public, important, or non-important place. Anorientation of the wearable extended reality appliance may provideinformation regarding whether the user is moving, a direction in whichthe user is moving, and whether the user is moving toward anotherlocation that might require another type of display mode. A location ofthe wearable extended reality appliance may include an area, district,locale, neighborhood, part, point, region, site, or any other indicationof a position of the wearable extended reality appliance. The locationof the wearable extended reality appliance may be determined based onuser input or data from any device capable of detecting information thatcan be used to determine such a position. For example, the environmentalcontext maybe determined to be the home of the user based on a specificaddress derived from Global Positioning System (GPS) data acquired froma GPS sensor. In another example, the same environmental context of thehome of the user may be determined based on a user input of the address,such as the user typing the address into a keyboard associated with theat least one processor. An orientation of the wearable extended realityappliance may include a direction, angle, location, tilt, slope, or anyother indication of a relative position of the wearable extended realityappliance. The orientation of the wearable extended reality appliancemay be determined based on user input or data from any device capable ofdetecting information that can be used to determine such a relativeposition. For example, the environmental context may be determined to besunny based on an eastern direction determined from GPS data acquiredfrom a GPS sensor. In another example, the same sunny environmentalcontext of the eastern direction may be based on a user input, such asthe user typing that the user is facing a sunny direction into akeyboard associated with the at least one processor. A current time mayinclude a date, day, hour, second, month, occasion, season, term, week,year, duration, interval, span, or any other indication of a presenttime. The current time may be determined based on user input or datafrom any device capable of detecting information that can be used todetermine such a present time. For example, the environmental contextmaybe determined to be 8 PM based on a measurement of light by anoptical sensor or a clock measurement. In another example, the sameenvironmental context of 8 PM may be based on a user input, such as theuser typing that the time is 8 PM into a keyboard associated with the atleast one processor.

In some embodiments, the environmental context is determined based on ananalysis of at least one of image data captured using an image sensorincluded in the wearable extended reality appliance or audio datacaptured using an audio sensor included in the wearable extended realityappliance. An image sensor may include a charge coupled device (CCD),complementary metal oxide semiconductor (CMOS), or any other device thatdetects and conveys information related to an image. Image data mayinclude pixel information, image size, resolution, depth of field, typeof objects, number of objects, and any other information that may berelated to an image. In one example, a visual classification algorithmmay be used to classify the image data to one of a plurality ofalternative classes. Each class may correspond to an environmentalcontext, and thereby the classification of the image data may determinethe environmental context. In another example, the image data may beanalyzed using an object detection algorithm to detect the present ofobjects of particular types in the environment, and the environmentalcontext may be based on whether objects of certain types are present inthe environment. The environmental context may be based on the imagedata using predefined parameters of image data for a given environmentalcontext, database values of image data used for a given environmentalcontext, or any other means of electronically linking one value withanother. For example, the environmental context may be determined to bea city based on the number of buildings detected in an image. In anotherexample, the environmental context may be determined to be night timebased on darkness measured from light data detected in an image. In yetanother example, the environmental context may be determined to be aprivate location based on a low-light condition, as measured by lightdata detected in an image. An audio sensor may include an acousticsensor, pressure sensor, microphone, or any other device that detectsthe presence or intensity of sound waves and converts it into electricalsignals. Audio data may include pressure, volume, tone, pitch, or anyother indication of any information associated with a sound. Theenvironmental context may be based on the audio data using predefinedparameters of audio data for a given environmental context, databasevalues of audio data used for a given environmental context, or anyother means of electronically linking one value with another. Forexample, the environmental context may be determined to be a publiclocation based on a high volume or sound level, as detected by amicrophone. In another example, the environmental context may bedetermined to be a private location, based on a low volume or soundlevel, as detected by a microphone. In yet another example, theenvironmental context may be determined to be a concert based on musicdetected in the audio data.

In some embodiments, the environmental context may be based on at leastone action of at least one individual in an environment of the wearableextended reality appliance. An action may include walking, standing,sitting, reading, speaking, singing, running, or any other process ofdoing something to achieve an aim. An individual in an environment ofthe wearable extended reality appliance may include any person within aspecified distance of the wearable extended reality appliance. It may bedesirable to base the environmental context on such an individual toprotect the privacy of certain information that a user of the wearableextended reality appliance may not want to share with others. It mayalso be desirable to base the environmental context on such anindividual to improve processor efficiency and speed by only associatingcertain display modes with certain individuals. For instance, someindividuals may rarely, if ever, need to know health information. Thismay include co-workers or shop attendants. Therefore, it would beinefficient to associate health-related display modes with a givenenvironmental context based on the presence of such an individual, andit may be desirable to only associate health-related display modes witha given environmental context based on the presence of a doctor, nurse,or any other individual who would need that type of information. Forexample, the processor may be configured such that an environment of thewearable extended reality appliance is any area within a ten-foot radiusof the wearable extended reality appliance. In this example, only theactions of people within that ten-foot radius may be considered todetermine an environmental context. In another example, some individualsmay be identified to have access to certain sensitive information in agiven environment. When those individuals are in an environment of thewearable extended reality appliance, more public display modes such asimproved brightness and contrast, may be used. When other individualsnot including those identified individuals are in the environment, moreprivate display modes, such as reduced brightness and contrast, may beused.

In some embodiments, the environmental context may be based on objectsin an environment of the wearable extended reality appliance. Objects inan environment of the wearable extended reality appliance may includeany item or surface within a specified range of the wearable extendedreality appliance. Basing the environmental context on such objects maybe desirable because certain objects may be associated with specificlocations. For instance, desks may be associated with workplaces, whiletrees may be associated with parks. For example, the processor may beconfigured such that an environment of the wearable extended realityappliance is any area within a five-foot radius of the wearable extendedreality appliance. In this example, only objects within that five-footradius may be considered to determine an environmental context. When theuser is in an environment with home objects such as kitchen appliances,plants, photographs, clothes, sofas, or beds, the environmental contextmay be determined to be a home environment. In such an example, theavailable display modes may include those associated with privatelocations, such as a home. Such display modes may include those havingincreased brightness and contrast, or unobscured text. When the user isin an environment with shop objects such as a cash register, clothesracks, display tables, or jewelry, the environmental context may bedetermined to be a shop environment. In such an example, the availabledisplay modes may include those associated with public locations, suchas a shop. Such display modes may include those having reducedbrightness and contrast, or obscured text.

Some embodiments may include a method for coordinating virtual contentdisplay with mobility status. FIG. 53 is a flowchart of an exemplarymethod 5300 of coordinating virtual content display with mobilitystatus, consistent with some embodiments of the present disclosure.Method 5300 may include a step S310 of accessing rules associating aplurality of user mobility statuses with a plurality of display modesfor presenting virtual content via a wearable extended realityappliance. Method 5300 may include a step S312 of receiving first sensordata from at least one sensor associated with the wearable extendedreality appliance, the first sensor data being reflective of a mobilitystatus of a user of the wearable extended reality appliance during afirst time period. Method 5300 may include a step S314 of based on thefirst sensor data, determining that during the first time period theuser of the wearable extended reality appliance is associated with afirst mobility status. Method 5300 may include a step S316 ofimplementing at least a first accessed rule to generate a first displayof the virtual content via the wearable extended reality applianceassociated with the first mobility status. Method 5300 may include astep S318 of receiving second sensor data from the at least one sensor,the second sensor data being reflective of the mobility status of theuser during a second time period. Method 5300 may include a step S320 ofbased on the second sensor data, determining that during the second timeperiod the user of the wearable extended reality appliance is associatedwith a second mobility status. Method 5300 may include a step S322 ofimplementing at least a second accessed rule to generate a seconddisplay of the virtual content via the wearable extended realityappliance associated with the second mobility status, wherein the seconddisplay of the virtual content differs from the first display of thevirtual content.

Some embodiments may include a system for coordinating display ofvirtual content with mobility status, the system comprising: at leastone processor configured to: access rules associating a plurality ofuser mobility statuses with a plurality of display modes for presentingvirtual content via a wearable extended reality appliance; receive firstsensor data from at least one sensor associated with the wearableextended reality appliance, the first sensor data being reflective of amobility status of a user of the wearable extended reality applianceduring a first time period; based on the first sensor data, determinethat during the first time period the user of the wearable extendedreality appliance is associated with a first mobility status; implementat least a first accessed rule to generate a first display of thevirtual content via the wearable extended reality appliance associatedwith the first mobility status; receive second sensor data from the atleast one sensor, the second sensor data being reflective of themobility status of the user during a second time period; based on thesecond sensor data, determine that during the second time period theuser of the wearable extended reality appliance is associated with asecond mobility status; and implement at least a second accessed rule togenerate a second display of the virtual content via the wearableextended reality appliance associated with the second mobility status,wherein the second display of the virtual content differs from the firstdisplay of the virtual content.

Docking virtual objects to physical objects may provide importantcapabilities in an extended reality environment. For example, the dockedvirtual objects may provide information related to a physical object,and/or may extend or adjust functionality of a physical object that hassome functionality. Docking the virtual objects to the physical object,and thereby causing the virtual object to move with the physical objectunder certain conditions, may maintain a spatial association or relationbetween the physical object and the virtual objects. Furthermore, inembodiments where the physical object is an input device (such as akeyboard), and the docked virtual objects are configured to provideusers with additional ways to provide input, maintaining the spatialassociation between the physical object and the virtual objects may benecessary for enabling touch typing (also known as blind typing), orother kinds of interactions based on constant special relations amongelements. Some virtual objects may have functionality that depend ontheir adjacentness to a physical surface. For example, a physicalsurface may provide tactile feel to a user using a virtual slider.Therefore, when removing the physical object from a physical surfaceassociated with virtual objects adjacent to the physical surface anddocked to the physical object (at least while the physical object ismoving on the surface), it may be desirable to adjust the virtualobjects to function away from a physical surface, to undock from thephysical object (for example, a wearable extended reality appliance usermay leave the virtual objects adjacent to the physical surface), and/orto remove the virtual objects from the extended reality environment.

Some disclosed embodiments may include systems, methods, andnon-transitory computer readable media, containing instructions thatwhen executed by at least one processor cause the at least one processorto perform operations for modifying display of virtual objects docked toa movable input device. Modifying display of virtual objects may includechanging visual properties such as color scheme, opacity, intensity,brightness, frame rate, display size, and/or virtual object type.Additionally, some virtual objects may be displayed and others may behidden depending on the location of the movable input device, i.e.,whether the input device is on a supporting surface or not.

A movable input device may refer to any input device that may be moved,for example by a user of an extended reality environment, such as avirtual or a physical device that a user of a wearable extended realityappliance may employ to send a command to the wearable extended realityappliance. The movable input device may be any portable device that auser can easily (i) move on a supporting surface, and/or (ii) remove andtransfer from the supporting surface to a different location ororientation. In some examples, the movable input device may be aphysical movable input device, such as a physical keyboard, a physicalcomputer mouse, a physical touchpad, a physical joystick, and/or aphysical game controller. In some examples, the movable input device maybe a virtual movable input device, such as a virtual keyboard, a virtualslider, and/or a combination or a virtual keyboard and a virtual slider.By way of example, as illustrated in FIG. 2, input devices such askeyboard 104 or mouse 106 may be movable input devices. Keyboard 104 andmouse 106 are movable input devices because each can be moved from itscurrent location or orientation to a new location or orientation. Forexample, keyboard 104 or mouse 106 may be moved towards the left orright from its current location. In another example, keyboard 104 may berotated about an axis perpendicular to the figure such that keyboard 104is oriented at an angle of, for example, thirty degrees from its initialorientation. In yet another example, keyboard 104 may be removed fromthe top surface of table 102. In an additional example, keyboard 104 maybe moved on the top surface of table 102 to a new location on the topsurface of table 102.

Some disclosed embodiments may include receiving image data from animage sensor associated with a wearable extended reality appliance, andthe image data may represent an input device placed at a first locationon a supporting surface.

An image sensor may be included in any of the devices or systems in thepresent disclosure and may be any device capable of detecting andconverting optical signals in the near-infrared, infrared, visible,and/or ultraviolet spectrums into electrical signals. Examples of imagesensors may include digital cameras, phone cameras, semiconductorCharge-Coupled Devices (CCDs), active pixel sensors in ComplementaryMetal-Oxide-Semiconductor (CMOS), or N-type metal-oxide-semiconductor(NMOS, Live MOS). The electrical signals may be used to generate imagedata. Consistent with the present disclosure, the image data may includepixel data streams, digital images, digital video streams, data derivedfrom captured images, and data that may be used to construct one or more3D images, a sequence of 3D images, 3D videos, or a virtual 3Drepresentation.

Image data may be used to represent where the input device has beenplaced on the supporting surface. For example, the image sensor maydetermine, based on captured image data, where the movable input deviceis located, and what portion of the surrounding area is a supportingsurface. Color data, as captured by the image sensor via a Bayer-filtersensor, a Foveon X3 sensor, and/or a 3CCD sensor, may be used todetermine which portion of the image is the movable input device, andwhich portion of the image is the supporting surface. In someembodiments, the image sensor may have a built-in processing unit, whichis capable of machine learning. In this embodiment, the image sensorcompiles and stores image data over time and may determine which portionof the image data is a keyboard and which is a supporting surface fromthe compiled data. In some embodiments, the image sensor may alsocapture image data and later send that data to the at least oneprocessor associated with the wearable extended reality appliance to beanalyzed further, i.e., determine whether the movable input device hasbeen moved to second location as part of the support surface, or hasbeen removed from the support surface.

A supporting surface may be any solid surface on which a user may placea movable input device, such as a keyboard, a mouse, or a joystick. Inone example, the supporting surface may be a flat, even surface. Inother examples, the supporting surface may be an uneven surface and mayinclude defects, cracks, and/or dust. Examples of supporting surfacesmay include a surface of a desk, a table, a floor surface, or any othersurface of an object on which the movable input device may be placed.

In some embodiments, the image sensor may be included in the wearableextended reality appliance. An image sensor may be said to be includedin the wearable extended reality appliance if at least a portion of theimage sensor or its associated circuitry is located within a housing ofthe appliance, is located on the appliance, or is otherwise connectableto the appliance. Some types of sensors that may be included in thewearable extended reality appliance are digital cameras, phone cameras,semiconductor Charge-Coupled Devices (CCDs), active pixel sensors inComplementary Metal-Oxide-Semiconductor (CMOS), or N-typemetal-oxide-semiconductors (NMOS, Live MOS). Light may be captured andconverted to electrical signals, which may be used to generate imagedata. In order to accurately capture image data, the image sensor may beconnected to the lens of the wearable extended reality appliance. Theimage sensor may also be part of the bridge of the wearable extendedreality appliance connecting the two lenses together. In thisembodiment, the image sensor may be able to accurately capture imagesbased on where the user of the wearable extended reality appliance islooking.

In some embodiments, the image sensor may be included in the inputdevice connectable to the wearable extended reality appliance. An inputdevice may be any hardware configured to send information or controlsignals to the appliance. Non-limiting examples of input devices includekeyboards, touchpads, mice, dedicated controllers, or personal devices(cell phones, tablets, smart watches, or other wearables) paired withthe extended reality appliance. The types of sensors that may beincluded as part of the input device may be any of the sensors describedearlier in the specification. In some embodiments, the image sensor maybe included in the center of the input device, the edges of the inputdevice, or anywhere else along the length of the input device that maycapture accurate image data. An image sensor may detect changes inlight, i.e., when a user lifts the movable input device off thesupporting surface, that may reflect a removal movement. The imagesensor may also be included in a plane transverse to the moveable inputdevice. The image sensor may capture gestures, and/or different coloredimages to determine the location of the movable input device and thesupporting surface.

In some embodiments, the movable input device may include at leastthirty keys and a touch sensor, and excludes a screen configured topresent media content. The movable input device may include any numberof keys. For example, the movable input device may include 10, 20, 30,40, or any number of keys that may allow a user to provide one or moreinputs. In some embodiments, the input device may include thirty or morekeys that may include alphanumeric characters, directional keys,functional keys, keys adjusting brightness and sound, keys related toperforming shortcuts (e.g., copy and paste, cut and paste, or print),keys related to describing the alphanumeric characters (e.g.,exclamation point, apostrophe, period), and/or keys related toperforming functions (e.g., addition, multiplication, or equalssymbols). In some embodiments, the keyboard may also be configured forlanguages that do not use alphanumeric characters, such as those thatuse logographic characters.

In some embodiments, the input device may exclude a screen configured topresent media content. A screen configured to present media content mayinclude any type of display capable of presenting text, images,animation, or videos. Examples of screens configured to present mediacontent may include a computer monitor, a TV screen, a tablet, and/or aphone screen. In one embodiment, a screen configured to present mediacontent may have, but is not limited to, a screen size greater than 5centimeters, 10 centimeters, 25 centimeters, or 50 centimeters; a screenwith image resolution greater than 400×300 pixels, greater than 600×400pixels, or greater than 800×600 pixels, a screen capable of displayingmore than two colors, or a screen configured to play a video stream. Inthis embodiment, the movable input device does not include any aphysical screen configured to present media content fixedly connected toit. To present content, a user of the movable input device may use thewearable extended reality appliance to present a virtual object orconnect the movable input device to a separate standalone physicalscreen, for example, a television in proximity to the input device.

Some disclosed embodiments may involve causing the wearable extendedreality appliance to generate a presentation of at least one virtualobject in proximity to the first location. A virtual object may includea virtual inanimate object, virtual animate object, two-dimensionalvirtual object, three-dimensional virtual object, virtual input object,virtual widgets, or any of the other representations described herein asvirtual objects. A virtual object may also allow adjustment of visualproperties (for example, of virtual content, of the extended realityenvironment, of the virtual object, or of other virtual objects in theextended reality environment), such as a brightness scale, volumeadjustment bar, task bar, or navigation pane. The user of the wearableextended reality appliance may use gestures to interact with thesevirtual objects to modify the virtual content. Virtual content may bedisplayed via the wearable extended reality appliance, and virtualobjects may be part of the virtual content or may be associated with thevirtual content. For example, virtual content may be a document or avideo. Virtual objects as part of (or associated with) that specificvirtual content may be volume or brightness settings, or a navigationpane for the document. Furthermore, a virtual object may betwo-dimensional or three-dimensional. More than two virtual objects maybe paired with one another. For example, one virtual object may be avideo. Two additional virtual objects that may be paired with thisvirtual object may be a brightness adjustment bar and/or a volumeadjustment bar.

Proximity may refer to a distance between the movable input device andthe presented virtual object. The virtual object may be presentedadjacent to the movable input device so that the user of the wearableextended reality appliance can easily view and edit the virtual object.The distance between the input device and the presented virtual object,i.e., the proximity between the presented virtual object and the movableinput device, may be configured by the user. For example, one user mayprefer to present virtual content further away from the movable inputdevice, such as at a distance of half a meter, whereas other users mayprefer to present virtual content closer to the movable input device,such as at a distance within twenty centimeters. In some examples, thevirtual object may be presented adjacent to the movable input device, ata selected relative position and orientation to the moveable inputdevice, so that the user of the wearable extended reality appliance caneasily locate and interact with the virtual object even when not lookingat the virtual object (for example, in a touch typing usage) or when thevirtual object is not presented to the user (for example, when it isoutside the field of view of the wearable extended reality appliance).

Some disclosed embodiments may involve docking the at least one virtualobject to the input device. Docking may refer to associating a virtualobject with the input device. This may occur by connecting a virtualobject to a physical object such that the two move together in tandem.That is, the virtual object's position and/or orientation may beconnected to the position of the other object position and/ororientation. In one example, a virtual display screen (also referred toas a ‘virtual display’ or a ‘virtual screen’ herein) may be docked to aphysical keyboard and moving the physical keyboard may cause the dockeddisplay screen to move. In another example, a virtual input element maybe docked to a physical keyboard adjacent to a physical surface that thephysical keyboard is placed on, and moving the physical keyboard on thephysical surface may cause the docked virtual input element to move onthe physical surface.

In some embodiments, virtual objects may be docked with other virtualobjects. In some examples, virtual objects may be two-dimensional. Somenon-limiting examples of such two-dimensional virtual objects mayinclude a task pane, volume or brightness adjustment bar, or document.In some other examples, virtual objects may be three-dimensional. Somenon-limiting examples of such three-dimensional objects may include apresentation or scale model of a project. In one example,two-dimensional virtual objects may be docked with three-dimensionalvirtual objects and vice-versa. In another example, two-dimensionalvirtual objects may be docked with two-dimensional virtual objects,and/or three-dimensional virtual objects may be docked withthree-dimensional virtual objects.

In some examples, a user of a wearable extended reality appliance mayundock a virtual object from a physical object or undock a virtualobject from another virtual object. For example, the user may do so byissuing a textual command via the movable input device, by issuing avoice command, by making a hand gesture, using a virtual cursor, or byany other means. For example, the user may undock by performing anaction that may be captured by an image sensor. The image sensor may beassociated with the wearable extended reality appliance and may belocated on either lens, the bridge connecting the two lenses of thewearable extended reality appliance, or any other location where thesensor may capture accurate image information. The image sensor may alsobe included in the input device connectable to the wearable extendedreality appliance. The action may be recognized in the image datacaptured using the image sensor, for example by analyzing the image datausing a visual action recognition algorithm, and the undocking may betriggered based on the recognition of the action.

By way of example, FIG. 54 depicts a virtual object 5420 (for example, avolume adjustment bar or brightness adjustment bar) docked to a movableinput device 5410. Here, an image sensor associated with the wearableextended reality appliance 5414 may detect, through captured image data,that input device 5410 is placed at a first location on support surface5416. In one example, user 5418 of wearable extended reality appliance5414 may initiate a command to dock virtual object 5420 to the movableinput device 5410, and as a result virtual object 5420 may be presentedin proximity to the first location. Because the movable input device5410, and virtual object 5420, are docked, the presented virtual objectmay move in tandem with the movable input device. A second virtualobject 5412, such as a virtual display screen, may also be displayed inaddition to first virtual object 5420. In one example, second virtualobject 5412 may also be docked to movable input device 5410 and maytherefore move in tandem with the movable input device (although secondvirtual object 5412 may be far from the first location). In anotherexample, second virtual object 5412 may not be docked to movable inputdevice 5410, and therefore movement of movable input device 5410 may nottrigger an automatic movement of second virtual object 5412.

In some embodiments, processor operations may further include docking afirst virtual object to the input device, the first virtual object beingdisplayed on a first virtual plane overlying the support surface. Avirtual plane may refer to a flat surface (on which a straight linejoining any two points on it may wholly lie) or to a non-flat surface.In this context, all content displayed on a virtual plane may bedisplayed in association with the surface, e.g., the content may appearflat to a viewer or may appear to be associated with a common surface.Alternatively, a virtual plane may be a curved surface and content maybe displayed along the plane of the curved surface. A virtual plane maybe considered virtual if it is employed in connection with an extendedreality display or an extended reality environment, regardless ofwhether the plane is visible. Specifically, a virtual plane may bedisplayed in color or with texture so that it may be visible to a wearerof an extended reality appliance, or the virtual plane may be invisibleto the eye, but might become perceptible when visible objects arelocated in the virtual plane. In one example, a virtual plane may beillustrated with virtual grid lines in an extended reality environment.A virtual plane may include, for example, a flat surface, a non-flatsurface, a curved surface, or a surface having any other desiredconfiguration. The first virtual plane may overlie the support surface,that is, it may appear on top of or transverse to the support surface.In one example, the first virtual plane may be parallel to or nearlyparallel to the support surface, meaning that there is a zero-degreeangle or a small angle (for example, below 1 degree, below 5 degrees,below 10 degrees, and/or below 20 degrees) between the virtual plane andthe support surface. A virtual object, such as a volume or brightnessadjustment bar, may be docked with the movable input device such thatthe movable input device is placed on the support surface, the firstvirtual plane overlies the support surface, and the first virtual planeand the virtual object may be in the same plane.

In some embodiments, processor operations may further include docking asecond virtual object to the input device, wherein the second virtualobject is displayed on a second virtual plane transverse to the firstvirtual plane. A second virtual plane is a plane that may be presentedat a different angle to the first virtual plane and the supportingsurface. The second virtual plane may be displayed transverse to thefirst virtual plane so that presented content is more visible to theuser of the wearable extended reality appliance. Additionally, becausethe first and second virtual objects are docked, the two virtual objectsmay move together. In this embodiment, the second virtual object may bedisplayed at a right angle or at a non-zero angle (such as an anglebetween 10 and 170 degrees, between 30 and 150 degrees, between 45 and135 degrees, between 60 and 120 degrees, between 85 and 95 degrees,between 46 and 89 degrees, and/or between 91 and 134 degrees) to thefirst displayed content. For example, a first virtual object, a volumeor brightness adjustment bar or navigation pane, may be presented by thewearable extended reality appliance. The second virtual object, such asa virtual display screen, may be displayed transversely so that it maybe easier for the user of the wearable extended reality appliance tointeract with the first virtual object and to better view the secondvirtual object.

By way of example, FIG. 54 shows an example of a second virtual object,a virtual screen presenting virtual content 5412, presented in one planeand an example of a first virtual object, volume or brightnessadjustment bar, presented in another plane transverse to the firstplane.

Some disclosed embodiments may include determining that the input deviceis in a second location on the support surface. Determining that theinput device is in the second location may involve using image analysisto detect the input device and the second location and determining thatthey correspond. In another example, determining that the input deviceis in the second location may involve using visual object detectionalgorithm to analyze image data to detect the input device at aparticular location on the support surface, thereby identifying thesecond location on the support surface. The at least one processor,based on image data captured by the image sensor, may determine that theinput device is in a second location on the support surface. Forexample, the processor, based on data captured by the image sensor, maydetermine that colors and/or textures in the area surrounding themovable input device have not changed, indicating that the movable inputdevice is still located on the support surface. In some embodiments, thesecond location on the support surface may refer to a location thatdiffers from the first location, but is still in the same plane on thesame support surface. For example, the support surface may be a desk,table, floor surface, or other flat or non-flat solid surface. In thisexample, the second location may be one that is directly right or left(or at another particular direction) of the first location. The at leastone processor, based on image data captured by the image sensor includedin the wearable extended reality appliance, may determine that themovable input device has changed location via analysis of captured imagedata, for example using a visual object tracking algorithm.

The image sensor may also be included in the input device connectable tothe wearable extended reality appliance. In some embodiments, the imagesensor may be located within the movable input device. In someembodiments, the processor associated with the wearable extended realityappliance may determine that the movable input device has moved from afirst location to a second location based on an analysis of image datacaptured using the image sensor included in the movable input deviceusing an ego-motion algorithm.

In another embodiment, the image sensor may be included in a planetransverse to the movable input device. In this example, the imagesensor may capture color, light, texture or defining features that mayassist in describing the environment surrounding the movable inputdevice. In this embodiment, the image sensor may capture sequentialimages. These images will differ from one another, for example (at leastpartly) due to the user is moving the movable input device from a firstlocation on a supporting surface to a second location. The sequentialimages may be analyzed using an object tracking algorithm to determinethat the input device moved to the second location on the supportsurface.

In some embodiments, determining that the input device is in a secondlocation on the support surface may be based on a position of the inputdevice determined using one or more positioning sensors included in theinput device. In some embodiments, determining that the input device isin a second location on the support surface may be based on a movementof the input device from the first location. In some examples, themovement of the input device may be determined using one or moremovement sensors included in the input device, such as motion sensor 373(see FIG. 3).

In some embodiments, processor operations may further include detectingat least one of a movement of the input device on the support surface ora removal movement of the input device from the support surface based onan analysis of the image data. In some examples, visual classificationalgorithm may be used to analyze image data received from an imagesensor and classify the image data to one class of three alternativeclasses, ‘no movement of the input device’, ‘movement of the inputdevice on the support surface’ and ‘removal movement of the input devicefrom the support surface’, thereby detecting the movements. In oneexample, the visual classification algorithm may be a machine learningclassification algorithm trained using training examples to classifyimages and/or videos to one of the three classes. An example of suchtraining example may include a sample image and/or a sample video of asample input device, together with a label indicating to which of thethree classes the training example corresponds. In some examples, avisual object tracking algorithm may be used to analyze image datareceived from an image sensor external to the input device (for example,in the wearable extended reality appliance) and to track the inputdevice in the image data to thereby obtain motion information. Further,the motion information may be analyzed to determine whether the imagedata corresponds to no movement of the input device (for example,integral of the movement vectors is zero or small), to a movement of theinput device on the support surface (for example, integral of themovement vectors is on or parallel to the support surface), or to aremoval movement of the input device from the support surface (forexample, integral of the movement vectors is at a non-zero angle to thesupport vector). In some examples, an ego-motion algorithm may be usedto analyze image data received from an image sensor included in theinput device and determine the motion of the input device. Further, themotion may be analyzed to determine whether the image data correspondsto no movement of the input device (for example, integral of themovement vectors is zero or small), to a movement of the input device onthe support surface (for example, integral of the movement vectors is onor parallel to the support surface), or to a removal movement of theinput device from the support surface (for example, integral of themovement vectors is at a non-zero angle to the support vector).

In some embodiments, processor operations may further include detectingat least one of a movement of the input device on the support surface ora removal movement of the input device from the support surface based onan analysis of motion data received from at least one motion sensorassociated with the input device. The movable input device may includeits own motion sensor that outputs data and from which a determinationof movement may be made. Further, the motion information may be analyzedto determine whether the motion corresponds to no movement of the inputdevice (for example, the integral of the movement vectors is zero orsmall), to a movement of the input device on the support surface (forexample, the integral of the movement vectors is on or parallel to thesupport surface), or to a removal movement of the input device from thesupport surface (for example, the integral of the movement vectors is ata non-zero angle to the support vector).

Some disclosed embodiments may involve determining that the input deviceis in the second location, and subsequently updating the presentation ofthe at least one virtual object such that the at least one virtualobject appears in proximity to the second location. Determining that theinput device is in the second location may be accomplished in a mannersimilar to the previously described determination associated with thefirst location. For example, at least one processor associated with thewearable extended reality appliance may determine that the movable inputdevice is in a second location based on captured image data, capturedmotion data, or both. In one example, based on captured image dataand/or motion data, the at least one processor may determine that themovable input device is in a second location and no longer moving. Inanother example, based on captured image data and/or motion data, the atleast one processor may determine that the movable input device is in asecond location and continues to move. After the processor determinesthat the input device is in a second location on the support surface,the wearable extended reality appliance may continue to present thevirtual content it was presenting at the first location. In one example,the presented virtual content may not appear until the movable inputdevice is stationary at a second location. For example, when a userwishes to slide the movable input device across a supporting surface,the virtual object may not be presented until the input device isstationary at the second location, as determined by the image sensorand/or the motion sensor. In another example, the presented virtualcontent may appear to be moving with the movable input device from aposition in proximity to the first location to a position in proximityto the second location. In yet another example, the presented virtualcontent may continue to appear in proximity to the first location untilthe movable input device is stationary at a second location. In someexamples, a user may configure the wearable extended reality applianceto begin to present the virtual object when the motion sensor as part ofthe movable input device, or the image sensor as part of the wearableextended reality appliance detects that the input device is slowing downor nearing a standstill. The at least one processor, based on capturedimage data, may determine that the input device is slowing down ornearing a standstill by comparing sequential images to one another. Whenthe processor determines that the movable input device is nearing astandstill, the processor may resume presenting virtual content from thefirst location. In the motion sensor example, the processor, based oncaptured position and/or orientation data from the motion sensor, maydetermine that the movable input device is nearing the second locationif the angle between the movable input device and the support surface isnearing zero. The wearable extended reality appliance may presentcontent at the second location when the angle between the bottom of themovable input device and the support surface is less than five degrees.

By way of example, in FIG. 55A, a user of a wearable extended realityappliance may move the input device 5510 from a first location 5512,where it is docked to a first virtual object 5514 on support surface5516 to second location 5518. A second virtual object 5520, positionedaway of support surface 5516, may also be docked with input device 5510.For example, the second virtual object 5520 may be comprised of a graphor chart, and the first virtual object 5514 may be an interactivevirtual tool used to adjust the brightness of the graph or chart. Asshown in FIG. 55B, the wearable extended reality appliance, in responseto determining that the input device 5510 is in a second location 5518,updates the virtual object presentation such that the first virtualobject 5514 appears in proximity to the second location 5518 on supportsurface 5516, and second virtual object 5520 moves with the input device5510 away of support surface 5516. In some examples, other virtualobjects (not shown in FIGS. 55A and 55B), such as virtual widgets, maybe docked to second virtual object 5520, and may thereby move with thesecond virtual object 5520 when the second virtual object 5520 moves.

In some embodiments, wherein the input device is placed on the firstlocation on the supporting surface, and the at least one virtual objecthas original spatial properties relative to the input device, processoroperations may further include maintaining the original spatialproperties of the at least one virtual object relative to the inputdevice where the input device is in the second location. Spatialproperties may include any characteristic or parameter reflective of theobject's position in an environment. For example, two objects or itemsmay have spatial properties relative to each other. When one of the twoobjects is moved, there may be a desire to maintain the same specialproperties for the other item, such that the second item moves with thefirst and in the new orientation, the relative spatial properties remainsubstantially the same. In one example, this may enable a user usingtouch typing (also known as blind typing) to use interactive virtualtool (e.g., first virtual object 5514) without looking at it, based onthe spatial properties of virtual object 5520 relative to the inputdevice 5510. In some other examples, a user of a wearable extendedreality appliance may move an input device from a first location on asupporting surface, such as a desk or table, to a second location on thesame supporting surface. When the movable input device is at or near thesecond location, a virtual object may be presented, and may be comprisedof a first virtual object docked with a second virtual object. Thepresented virtual object's properties may be configured by the user, andmay include color scheme, opacity, intensity, brightness, frame rate,display size, and/or virtual object type settings. When presentingvirtual content, a main goal may be to maintain the uniformity of thepresented virtual content when moving the input device from location tolocation. To achieve this goal, the spatial properties of the presentedvirtual object may remain the same.

In some embodiments, the original spatial properties may include atleast one of a distance of the at least one virtual object from theinput device, an angular orientation of the at least one virtual objectrelative to the input device, a side of the input device on which the atleast one virtual object is located, or a size of the at least onevirtual object relative to the input device. The original spatialproperties may include a distance of the at least one virtual objectfrom the input device. It may be desirable to keep the presented virtualcontent and the associated virtual objects at the same distance from theinput device at the second location as it was in the first location. Theuser may configure his or her preferred presentation distance relativeto the movable input device. For example, a user may prefer to presentvirtual content twenty-five centimeters away from the movable inputdevice. When the user moves the input device from a first location to asecond location, it may be desirable for the user to maintain theoriginal spatial properties so that he or she does not have to getre-oriented with the presentation, and/or that other users do not getconfused with a changing presentation.

In some embodiments, the original spatial properties may include anangular orientation of the at least one virtual object relative to theinput device. It may be desirable to keep the presented virtual contentand the associated virtual objects at the same angular orientation atthe second location as it was at the first location. For example, it maybe disruptive if virtual content was presented at a completely differentangle at the second location as it was at the first location and mayconfuse other users of wearable extended reality appliances who areviewing the content, for example, if the content was displayed upsidedown.

In some embodiments, the original spatial properties may include a sideof the input device on which the at least one virtual object is located.It may be desirable to display the virtual content on the same side ofthe input device at the first location as the second location so thatusers do not have to reorient themselves when viewing presented content.If virtual content is displayed at a location behind the movable inputdevice, it may be disruptive and confusing to other users of otherwearable extended reality appliances viewing the content.

In some embodiments, the original spatial properties may include thesize of the at least one virtual object relative to the input device. Itmay be desirable to keep presented virtual content and associatedvirtual objects at the same size at the second support surface locationas they were at the first support surface location. For example, keepingthe presented virtual content at the same size between locations ensuresthat all text and graphics, as part of the virtual content, arepresented at the same size so that other viewers of the content are ableto clearly view and understand the presented content.

Some disclosed embodiments may include determining that the input deviceis in a third location removed from the support surface. The thirdlocation determination may be accomplished in a manner similar the priordescription of the first and second location. In one example, it may bedetermined that the third location is removed from the support surfaceby geometrically measuring the distance between the input device and thesupport surface. In some embodiments, the image sensor associated withthe wearable extended reality appliance may determine that the inputdevice has been removed from the support surface based on captured imagedata. In some embodiments, the motion sensor associated with the movableinput device may determine that the input device has been removed fromthe support surface.

In some embodiments, modifying the presentation of the at least onevirtual object in response to the determination that the input device isremoved from the support surface may include continuing to present theat least one virtual object on the support surface. The processorassociated with the wearable extended reality appliance may determinethat the user has removed the input device from the support surface viacaptured image data, captured motion data, or both. When the userremoves the input device from the support surface the virtual objectbeing presented by the wearable extended reality appliance may notdisappear. Instead, the virtual object may remain visible to the user ofthe wearable extended reality appliance. This may be useful insituations when the user wishes to move the input device but does notwant to cause any disruptions when presenting a virtual object. Forexample, the user may be moving the input device to a different locationso that the virtual object is more visible to other viewers. In thisexample, the user of the wearable extended reality appliance may desireto continue presenting the at least one virtual object without anydisruptions. In some embodiments, after the input device is removed fromthe support surface, the spatial properties of the presented virtualobject may remain identical to what they were before the input devicewas removed, or they may change.

In some embodiments, processor operations may further includedetermining a typical location of the input device on the supportsurface and presenting the at least one virtual object in proximity tothe typical location when the input device is removed from the supportsurface. The typical location may be a location where the user prefersto place the movable input device and may be determined, for example,based on historical position and orientation data captured by the motionsensor. The typical location may also be determined based on thehistorical location of the movable input device as captured by the imagesensor associated with the wearable extended reality appliance. Theimage sensor may, over time, capture image data representing a positionand/or orientation of the input device. Based on that captured imagedata, the at least one processor may be able to predict typicallocations on the support surface where the user of the wearable extendedreality appliance prefers to present a virtual object. In someembodiments, the at least one processor may use a machine learningalgorithm or similar algorithm to predict a typical location. Themachine learning embodiments may receive training data including, forexample, historical placement data and frequency with which the inputdevice is placed at each location. The machine learning algorithm may betrained using the training data. The trained machine learning model maydetermine a typical location for placement of the input device. Thetypical location or data relating to it may be stored in a datastructure and retrieved by the processor when needed. In someembodiments, the typical location may be manually configured by theuser. For example, if a user typically uses the wearable extendedreality appliance while at a desk or table in a conference room, thatuser may manually configure a particular part of that desk or table asthe typical location to present virtual content.

In response to the determination that the input device is removed fromthe surface, some embodiments may include modifying the presentation ofthe at least one virtual object. In one example, modifying thepresentation may include altering a location or orientation of a virtualobject, for example in relation to the input device or to anotherobject. In another example, modifying the presentation may includealtering an opacity, a color scheme or a brightness of the presentationof the at least one virtual object. For example, the opacity may bereduced, the brightness may be increased, and/or any other parameter maybe increased or decreased. In some examples, modifying the presentationmay include altering a size of the at least one virtual object. In otherexamples, modifying the presentation may include modifying theappearance of the at least one virtual object in any possible way. Theprocessor may determine that an input device has been removed from asupporting surface based on either captured image data and/or capturedmotion data. In response, the presented virtual object may be modified.Presentation of a virtual object may be modified in a myriad of ways byaltering one or more parameters associated with the virtual object. Suchparameters may include, for example, the color scheme, opacity,intensity, brightness, frame rate, display size, and/or virtual objecttype may be modified. Typically, a modification of one or more of theseparameters may involve reducing or increasing that parameter, e.g.,reducing or increasing brightness or display size. In another example,multiple virtual objects may be presented. Modifying the presentation ofthe virtual objects may involve removing one of the virtual objects, or,when the input device is removed from the support surface, undocking afirst and second virtual object from one another. In some examples,image data captured using an image sensor included in the wearableextended reality appliance may be analyzed to select a modification tothe presentation of the at least one virtual object. For example, theimage data may be analyzed to determine ambient illumination conditions,and the modification may be selected based on the determined ambientillumination conditions. In another example, the image data may beanalyzed to detect a presence of other people in the environment (forexample using a person detection algorithm), in response to adetermination that other people are present, a first modification may beselected (for example, reducing size and opacity), and in response to adetermination the no other person is present, a second modification maybe selected (for example, reducing opacity while increasing size).

By way of example, FIG. 56A illustrates moving the input device 5610from a second location 5612 on the supporting surface to a thirdlocation 5614 removed from the supporting surface 5616. Removing theinput device 5610 from the supporting surface may cause the at least onevirtual object to be modified. Here, first virtual object 5618 andsecond virtual object 5620 were previously docked to the input device.However, when the input device is removed from support surface 5616, thesecond virtual object 5620 may be reduced in display size, and firstvirtual object 5618 may undock from the input device and stay on supportsurface 5616.

In some embodiments, modifying the presentation of the at least onevirtual object in response to the determination that the input device isremoved from the support surface may include causing the at least onevirtual object to disappear. When a user of a wearable extended realityappliance wishes to move from place to place, it may be desirable forthe virtual object to disappear when the input device is removed fromthe supporting surface, for example, when the virtual object may nolonger be useful. In another example, the user may wish to continuepresenting the at least one virtual object, but may need to move toanother location far away from the first location to do so. Such alocation may include a different conference room, a colleague's office,or a different room in one's home. Some embodiments may allow the atleast one virtual object to disappear when the input device is removedfrom the supporting surface so that it may reappear when the inputdevice is placed on a different supporting surface at a later time.Additionally, causing at least one virtual object to disappear mayreduce battery consumption and heat generation in the wearable extendedreality appliance because the user is no longer actively presentingduring the time that the input device is being moved from one locationto another.

In another embodiment, when the user removes the movable input devicefrom the supporting surface, a first virtual object may become undockedfrom a second virtual object. When the objects become undocked from oneanother, the second virtual object may no longer automatically move fromlocation to location with the first virtual object. One or both of theobjects may disappear, depending on how the user of the wearableextended reality appliance configures the appliance. The user of thewearable extended reality appliance may also configure a virtual objectto disappear based on virtual object type. For example, the task bar,navigation pane, or volume or brightness bar may automatically disappearwhen the appliance user removes the input device from the supportingsurface.

By way of example, as illustrated in FIG. 56B, when movable input device5610 is removed from the support surface 5616, first virtual object 5618(shown in FIG. 56A) may disappear completely and second virtual object5620 may be displayed unaltered. In other embodiments, second virtualobject 5620 may disappear and first virtual object 5618 may stayvisible, or in further embodiments, both first virtual object 5618 andsecond virtual object 5620 may disappear.

In some embodiments, processor operations may further include receivinginput indicating a desire of a user of the wearable extended realityappliance to interact with the at least one virtual object while theinput device is in the third location and causing the at least onevirtual object to reappear. A user's desire to interact with the atleast one virtual object in the third location may occur in a myriad ofways. For example, the user of the wearable extended reality appliancemay issue a voice command to interact with the at least one virtualobject. Such a voice command may include commands such as “presentvirtual object,” “modify virtual object,” “undock virtual objects,” orany other command that suggests an interaction between the user of thewearable extended reality appliance and the and the virtual content. Thevoice command may be received by audio input 433 (see FIG. 4). Theunique voice command may be configured by the user of the wearableextended reality appliance and may differ between different users. Inanother embodiment, the appliance user may move a virtual cursor inorder to interact with the at least one virtual object. In this example,the at least one processor may recognize the virtual cursor movement asa desire to interact with the at least one virtual object. In lieu of avirtual cursor, a joystick input, touch pad input, gesture input, orgame controller input may be used. In some embodiments, the user of thewearable extended reality appliance may click on an icon to present theat least one virtual object. The cursor, joystick, touch pad, or gamecontroller may be configured as part of the input interface 430 (seeFIG. 4). In another embodiment, the desire to interact with the at leastone virtual object may be determined through captured image data via theimage sensor. This captured image data may involve hand movements orgestures, head nods or gestures, or any other body movement that maysignal a desire to interact with the at least one virtual object, andmay be configured by the user of the wearable extended realityappliance. The data may be gesture input 431 (see FIG. 4). In thisexample, an appliance user may wave his or her hand to signal an intentto interact with the at least one virtual object. In response, thevirtual object may reappear on the virtual screen in proximity to theuser of the wearable extended reality appliance. In another embodiment,the desire to interact with the at least one virtual object may be basedon data other than captured image, voice, or cursor data. For example,the wearable extended reality appliance may determine that a user maydesire to interact with the at least one virtual object based oncalendar data. Calendar data may refer to any data that is relevant to auser's tasks for the day. For example, a user may have multiple meetingsscheduled for the day. The user of the wearable extended realityappliance may need to present at least one virtual object at each ofthese meetings. The wearable extended reality appliance may be able todetermine based on the user's schedule that the user may desire tointeract with a virtual object. In this example, calendar data mayindicate that the user of the wearable extended reality appliance has ameeting on a Wednesday (or other workday), and that he or she needs topresent a pie chart or other graphical content during the meeting. Thewearable extended reality appliance, based on that data, mayautomatically present that content, and the user does not need tomanifest an additional intent to interact with the virtual object. Inanother example, the wearable extended reality appliance may determinethat the user wishes to interact with the at least one virtual objectbased on stored usage data. Stored usage data may refer to data thatshows one or more times at which the user has presented at least onevirtual object. The stored data may also include the day of the weekthat the user presented the at least one virtual object. Based on thestored usage data, the wearable extended reality appliance may be ableto predict when the user wishes to interact with the at least onevirtual object. The processor associated with the wearable extendedreality appliance may predict user interactions based on a machinelearning model or similar algorithm. In this embodiment, training data,including, for example, calendar data and information about one or morevirtual objects in association with the calendar data may be used totrain a machine learning model. The trained machine learning model maydetermine a user's intent to interact with a virtual object based on atime of day or other calendar information.

In some embodiments, modifying the presentation of the at least onevirtual object in response to the determination that the input device isremoved from the support surface may include changing at least onevisual property of the at least one virtual object. A visual propertymay include any characteristic impacting perception.

In some embodiments, the at least one visual property may include atleast one of a color scheme, an opacity level, an intensity level, abrightness level, a frame rate (when video content is presented), adisplay size, an orientation, and/or virtual object type. Any one ofthese or other visual properties may be increased, reduced or otherwisealtered. For example, a display size or brightness level may beincreased or reduced in response to the determination that the inputdevice has been removed from the support surface. By way of example,FIG. 56A shows a second virtual object 5620 reduced in display sizeafter the input device 5610 has been removed from support surface 5616.

The user of the wearable extended reality appliance may configure whichvisual property to modify based on the virtual object that is presented.In this embodiment, whenever a particular virtual object type (e.g.,document, video, or graph) is presented, it may be assigned to a defaultvisual property to be modified. For example, if a user is presenting achart or a graph, the user may decide to reduce the brightness in lieuof the orientation, such that a viewer may still understand the mainpoint of the chart of graph. In another example, if a user of a wearableextended reality appliance is presenting a photograph or drawing, theuser may decide to reduce display size instead depending on the user'spreference. In another embodiment, visual property to be modified may bebased off the previous modification the user made. For example, a usermay present a chart or a graph, and modify the brightness settings ofthose presented virtual objects. The visual data (e.g., a brightness orother visual setting) may be stored in a memory associated with thewearable extended reality appliance, input device, or a remote server.When the user decides to present similar content at a later time, the atleast one processor may adjust brightness before other visual propertiesbased on the visual data stored in the memory. In yet anotherembodiment, the at least one processor may predict which visual propertyto modify based on stored data, a machine learning model, or anothersimilar algorithm.

In some embodiments, modifying the presentation of the at least onevirtual object in response to the determination that the input device isremoved from the support surface may include presenting a minimizedversion of the at least one virtual object. A minimized version may be arepresentation that is altered in at least one dimension. Virtualobjects may be minimized when either the image sensor associated withthe wearable extended reality appliance, or the motion sensor associatedwith the movable input device detects that the input device is removedfrom the support surface. Minimized objects may refer to the virtualobjects in icon form. For example, a user may be presenting a textdocument in a text editor or a presentation in a presentation viewingapplication. When these documents are minimized, a viewing user willonly see the icon representing these programs displayed. Such virtualobjects may be minimized because the wearable appliance user is likelyno longer actively presenting, so the virtual objects may be minimizedin order reduce battery consumption and to reduce heat generation in thewearable extended reality appliance. Virtual objects may also beminimized when a user is not actively presenting to others, but is usinga program via a virtual screen. In one example, a user may privately beusing a messaging application to message colleagues. While not activelypresenting content to others, the user may be still using the movableinput device to interact with the virtual screen, and thus when themovable input device is removed, the messaging application will beminimized such that the user only sees the icon. In one example, the atleast one virtual object may be an application, such as a chatting appor a messaging app. When the input device is placed on the supportingsurface, the app may be opened, and the user may see recent chats andmessages. However, when the keyboard is removed from the supportingsurface, the application may be automatically minimized to an icon. Theuser may still be able to receive messages and notifications, but thecomplete messages may not be shown unless the input device is placed ona supporting surface. In another example, the virtual object may be adisplayed video. When the input device is removed from the supportingsurface, the video may be minimized and may automatically pause thevideo content. When the input device, after being removed from thesupporting surface, is placed back on the supporting surface, the videocontent may resume playing at the original display settings.

In some embodiments, processor operations may further include receivinginput reflective of a selection of the minimized version of the at leastone virtual object and causing the at least one virtual object to bepresented in an expanded view. An expanded view may include anypresentation where at least one dimension is increased. In someembodiments, the presented virtual object may be minimized after theinput device is removed from the supporting surface. The user of thewearable extended reality appliance may select the minimized virtualobject and present it in an expanded view. Additionally, inputreflective of a selection of the minimized version of the at least onevirtual object may arise in a myriad of ways. For example, the user maytype a command into input unit 202 (see FIG. 3), instructing theprocessor to expand the minimized application. In another example, theinput may be pointer input 331 (see FIG. 3) selecting the minimizedapplication and clicking on the application icon to expand it. In afurther example, input may be a voice command included in audio input433 (see FIG. 4) instructing the processor to expand the minimized icon.Such voice commands may be configured by the user of the wearableextended reality appliance. In yet another example, the input may be inthe form of captured image data via as gesture input 431 (see FIG. 4).In this example, the input may be in the form of a hand wave or gesture,a head nod, or any other bodily movement that the wearable extendedreality appliance may interpret as a command to expand the minimizedapplication.

FIG. 57 is a flow chart illustrating an exemplary method 5700 formodifying display of virtual objects docked to a movable input device,based on the movable input device's location. Method 5700 may beperformed by one or more processing devices (e.g., 360, 460, or 560)associated with input unit 202 (see FIG. 3), XR unit 204 (see FIG. 4),and/or input interface 330, 430 (see FIGS. 3 and 4). The steps of thedisclosed method 5700 may be modified in any manner, including byreordering steps and/or inserting or deleting steps. Method 5700 mayinclude a step S712 of receiving image data from the image sensor 472(see FIG. 4) associated with XR unit 204, the image data representing aninput device placed at a first location on a supporting surface. Method5700 may also include a step S714 of causing the wearable extendedreality appliance to generate a presentation of at least one virtualobject in proximity to the first location. The at least one processor460, as illustrated in step S716, may dock the at least one virtualobject to the input device, e.g., keyboard 104 (see FIG. 1). Method 700may include a step S718 that involves the at least one processorassociated with XR unit 204 (see FIG. 4) determining that the inputdevice (e.g., keyboard 104) is in a second location on the supportingsurface. The processor 460 (see FIG. 4) may perform a step S720 that, inresponse to a determination that the input device is in a secondlocation, involves updating the presentation of the at least one virtualobject such that the at least one virtual object appears in proximity tothe second location. Method 5700 may include a step S722 wherein theprocessor associated with the wearable extended realty appliance 204(see FIG. 4) determines that the input device is in a third locationremoved from the support surface. Method 5700 may include a step S724involving, in response to the determination that the input device isremoved from the support surface, modifying the presentation of the atleast one virtual object based on the third keyboard location.

Some disclosed embodiments may include systems, methods and/ornon-transitory computer readable media containing instructions that whenexecuted by at least one processor may cause the at least one processorto perform operations for docking virtual objects to virtual displayscreens in an extended reality environment. A virtual object may be anynon-physical item that appears in the extended reality environment. Thevirtual object may include a visual presentation rendered by a computerin a confined region and configured to represent an object of aparticular type (such as an inanimate virtual object, an animate virtualobject, virtual furniture, a virtual decorative object, virtual widget,or other virtual representation). For example, the virtual object mayinclude a widget, documents, presentations, media items, photos, videos,virtual characters, and other objects. The virtual object may appearwhen a user is wearing a wearable extended reality appliance. An exampleof a virtual display screen (also referred to as a “virtual display” ora “virtual screen” herein) may include virtual display screen 112described above, and may include a virtual object mimicking and/orextending the functionality of a physical display screen, as describedabove. An extended reality environment may be an environment that isaccessed by means of a computer. For example, the extended realityenvironment may be accessed when the user is wearing the wearableextended reality appliance. Docking virtual objects may includeconnecting, tethering, linking or otherwise joining at least twoobjects. A docked virtual object's position and/or orientation may beconnected to the position and/or orientation of another object'sposition and/or orientation. For example, docked virtual objects maymove together with the objects they are docked to. As another example, avirtual object may be docked to a virtual display screen and moving thevirtual display screen may cause the docked virtual object to move.

In some embodiments, the at least one virtual object may include adedicated widget virtually representing a phone of a user of thewearable extended reality appliance. A widget may be a module,application, or interface that allows a user to access information orperform a function, and appearing in a defined area of an extendedreality environment. A dedicated widget may be a module, application, orinterface (or an icon or other representation linked to the module,application or interface) that provides specific information or performsa specific functions. For example, a dedicated widget may be a phoneapplication that allows a user to perform the functions of a phone. Auser may be able to dial a phone number or create a text message usingthe dedicated widget. In one example, the widget may present a replicaof the phone display screen in the extended reality environment.

By way of example, FIG. 58 illustrates an example of a virtual displayand a docked virtual object representing a phone of a user, consistentwith some embodiments of the present disclosure. As illustrated in FIG.58, virtual object 5811 may be docked to virtual display 5810. Virtualobject 5811 may represent a phone of a user. For example, virtual object5811 may be or include a virtual widget configured to virtuallyrepresent a phone of a user of the wearable extended reality appliance.

Some disclosed embodiments may include generating virtual content forpresentation via a wearable extended reality appliance, where thevirtual content includes a virtual display and a plurality of virtualobjects located outside the virtual display. A wearable extended realityappliance may present virtual content to a user as discussed elsewherein this disclosure. The virtual content may include a virtual displayand one or more virtual objects that may or may not be in proximity tothe virtual display. For example, the virtual objects may be outside thevirtual display when the virtual objects are not in the same space asthe virtual display. When the virtual objects are located outside thevirtual display, the virtual objects may not move with the virtualdisplay. For example, when a user is wearing the wearable extendedreality appliance and the virtual content is presented to the user, theuser may move the location of the virtual display. When the user movesthe location of the virtual display, the location of the virtual objectsmay not change because the virtual objects are located outside thevirtual display.

By way of example, FIGS. 59A and 59B illustrate a virtual display and aplurality of virtual objects located outside the virtual display beforeand after the virtual display changes locations, consistent with someembodiments of the present disclosure. As illustrated in FIG. 59A,virtual objects 5911, 5912, and 5913 may be located at locations 5915,5916, and 5917 respectively. Virtual display 5910 may be located atlocation 5914. Virtual objects 5911 and 5912 and virtual display 5910may be located above dashed line 5918. Virtual display 5910 may not bedocked to virtual objects 5911, 5912, and 5913. As illustrated in FIG.59B, virtual display 5910 may move to a new location 5926 below dashedline 5918. Virtual objects 5911, 5912, and 5913 may not move withvirtual display 5910. Virtual objects 5911, 5912, and 5913 may stay intheir original locations 5915, 5916, and 5917, respectively, abovedashed line 5918.

Some disclosed embodiments may include receiving a selection of at leastone virtual object from the plurality of virtual objects. Receiving aselection of a virtual object may include determining, setting, fixing,or picking a virtual object. A user may pick at least one virtual objectfrom a plurality of virtual objects. The user may pick a virtual objectby clicking a mouse, pressing a button, tapping a touch surface,dragging an object, highlighting an object, through hand gesture,through voice commands, or by any other means. For example, a user maypick a virtual object that represents a document from the plurality ofvirtual objects. As another example, a user may pick a virtual objectthat represents an audio file from the plurality of virtual objects.Upon picking by the user, the act of picking may cause information to betransmitted to at least one processor, and the at least one processormay there receive the selection. In other examples, receiving theselection may comprise reading the selection from memory, receiving theselection from an external device, receiving the selection from ananalysis of data (such as an input data, image data, audio data, and soforth), determining the selection based on one or more rules, and soforth.

In some embodiments, the at least one virtual object selected from theplurality of virtual objects may include a first virtual objectdisplayed on a first surface and a second virtual object displayed on asecond surface that at least partially coincides with the first surface.In one example, the first and/or the second surfaces may include one ormore exterior surfaces of any physical object such as a table, a desk, acupboard, an end table, an arm of a chair, or any other object presentin the user's physical environment. In one example, the first and/or thesecond surfaces may be virtual surfaces. In another example, one of thefirst and/or the second surfaces may be a virtual surface, and the othermay be a physical surface. The first virtual object and the secondvirtual object may be projected onto different surfaces. For example,the first surface may be a physical surface that a keyboard is placed onand the second surface may be a virtual surface. The two surfaces mayintersect in a straight or a curved line.

In some embodiments, the first surface may be substantiallyperpendicular to a ground surface and the second surface may besubstantially parallel to the ground surface. In one example, a groundsurface may be any horizontal surface, relative to a gravity direction,at or near ground level. For example, a ground surface may be a surfacethat includes a floor. The first and second surface may be arrangedrelative to the ground surface. For example in some embodiments, thefirst surface may be oriented perpendicular to the ground surface,whereas the second surface may be oriented parallel to the groundsurface. In some embodiments, the first and second surface may each beinclined at a different angle relative to the ground surface.

Some disclosed embodiments may include changing plane cursor movementbetween the first and second surfaces. A cursor (also referred to as a‘virtual cursor’ herein) may include an indicator that appears in theextended reality display to display a selected position on the extendedreality environment, as described above. The cursor may identify a pointon the extended reality environment that may be affected by a userinput. Plane cursor movement may refer to a movement of the within atwo-dimensional plane in response to the user input. The cursor may movebetween a first and second surface. For example, the first surface maybe a top surface of a table and the second surface may be a virtualsurface.

Some disclosed embodiments may include docking the at least one virtualobject to the virtual display. At least one virtual object out of theone or more virtual objects located outside the virtual display may bedocked to the virtual display in the manner described earlier. Dockingof the one or more virtual objects to the virtual display may allow theone or more objects and the display to move together. For example, whena user moves the virtual display in the extended reality environment,the docked one or more virtual objects may move with the virtualdisplay. In some examples, docking a virtual object to the virtualdisplay may include adding the virtual object to a data-structure ofvirtual objects docked to the virtual display (such as a list, a set, adatabase, and so forth). When the virtual display moves, thedata-structure may be accessed to determine which virtual objects needsto be moved with the virtual display. In one example, undocking thevirtual object from the virtual display may include removing the virtualobject from the data-structure. In some examples, docking a firstvirtual object to a second virtual object may include adding the firstvirtual object to a data-structure of virtual objects docked to thesecond virtual object (such as a list, a set, a database, and so forth).When the second virtual object moves, the data-structure may be accessedto determine which virtual objects needs to be moved with the secondvirtual object. In one example, undocking a first virtual object from asecond virtual object may include removing the first virtual object fromthe data-structure.

By way of example, FIGS. 60A and 60B illustrate an example of a virtualdisplay and a plurality of virtual objects docked to the virtualdisplay. As illustrated in FIG. 60A, virtual display 5910 may be locatedat location 5914. Virtual objects 5911 and 5912 may be located atlocations 5915 and 5916, respectively. Virtual object 5913 may belocated at location 5917. Virtual objects 5911, 5912, and 5913 andvirtual display 5910 may be located above dashed line 5918. Virtualdisplay 5910 may be docked to virtual object 5912, and virtual object5912 may be docked to virtual object 5911, as illustrated in FIG. 60Awith a solid line connecting virtual display 5910 and virtual object5912 and a solid line connecting virtual objects 5912 and 5911. It isunderstood that these solid lines may not actually exist, but are onlyused to illustrate the docking. Such solid lines, or other visualindications of docking, may or may not be presented in the extendedreality environment. Virtual object 5913 may not be docked to any otherobject or to virtual display 5910. As illustrated in FIG. 60B, thelocation of virtual display 5910 may move to a new location 5926, belowdashed line 5918. Virtual objects 5911 and 5912 that are docked withvirtual display 5910 (directly, as in the case of virtual object 5912,or indirectly, as in the case of virtual object 5911 that is docked tovirtual object 5912 and therefore is indirectly docked to the virtualdisplay) may also move to new locations 6024 and 6022, respectively,below dashed line 5918. Virtual object 5913 that is not docked withvirtual display 5910 (directly or indirectly), however, may remain insame location 5917, above dashed line 5918.

In some embodiments, a duration of association between the at least onevirtual object and the virtual display may be time-dependent. Theduration of association between the at least one virtual object and thevirtual display may represent a period of time for which the at leastone virtual object may remain docked to the virtual display. The periodof time may be one or more seconds, minutes, hours, or any other lengthof time. For example, the at least one virtual object and the virtualdisplay may only be connected for thirty seconds. During the thirtyseconds, the virtual object and the virtual display may move together.After expiry of the thirty seconds, the virtual object may not move whenthe virtual display moves, and the virtual display may not move when thevirtual object moves. As another example, the at least one virtualobject and the virtual display may only be connected for five minutes.During the five minutes, the virtual object and the virtual display maymove together. After expiry of the five minutes, the virtual object maynot move when the virtual display moves, and the virtual display may notmove when the virtual object moves

Some disclosed embodiments may include moving the at least one virtualobject with the virtual display during a first time period, in responseto a change in the location of the virtual display during the first timeperiod and dissociating the at least one virtual object from the virtualdisplay during a second time period different from the first timeperiod, in response to a second change in the location of the virtualdisplay during the second time period. The at least one virtual objectand the virtual display may be docked together for a certain timeperiod. During the first time period, the at least one virtual objectand the virtual display may move together. When the first time period isover and the second time period begins, the at least one virtual objectand the virtual display may not be docked together. During the secondtime period, the at least one virtual object may not move with thevirtual display. For example, during the second time period, a user mayprovide an input to move the virtual display. The virtual display maymove based on the input and not move the at least one virtual object.The at least one virtual object may remain in the same location.

In some embodiments, docking the at least one virtual object to thevirtual display may open a communications link between the at least onevirtual object (or a module controlling the at least one virtual object,such as a software program, a computing device, a smartphone, a cloudplatform, etc.) and the virtual display (or a module controlling thevirtual display, such as an operating system, a computing device, acloud platform, etc.) for exchanging data, and wherein the operationsmay further include retrieving data from the at least one virtual object(or the associated module) via the communications link and displayingthe retrieved data on the virtual display. Docking may include creatinga communications link between at least two objects (or between theassociated modules). A communications link may be a connection betweentwo or more object (or between the associated modules) that may allow atransfer of information, data, or commands from one object to another.The virtual object and virtual display may transfer data between eachother via the communications link. The data associated with the virtualobject may be transferred to the virtual display and displayed to auser. The data may be audio, visual, text, or other types of data. Forexample, the virtual object may contain audio files. The audio files maybe transferred to the virtual display through the communication link.The audio files may be used to play audio in association with thevirtual display. As another example, the virtual object may containdocument files. The document files may be transferred to the virtualdisplay through the communication link. The document files may bedisplayed on the virtual display so a user can read, edit, or changeother functions of the document files. As another example, the virtualobject may contain image files that may be transferred to the virtualdisplay through the communications link and displayed on the virtualdisplay.

Some disclosed embodiments may include receiving an input indicative ofan intent to change a location of the virtual display without anexpression of an intent to move the at least one virtual object, afterdocking the at least one virtual object to the virtual display. A usermay decide to move the virtual display without exhibiting an intent tomove the virtual object. An intent of a user of the first wearableextended reality appliance to move the virtual display may be determinedbased on an input by the user. Such an input may include interactionwith one or more objects via a touch-screen interface, a mouse, agesture, a voice command, or any other type of interactive element. Forexample, the intent to move the virtual object may be determined whenthe user provides an input to change a location of the virtual display.The location may be determined relative to a set of coordinate axes, forexample, by using three coordinate values with reference to a set ofCartesian or polar coordinate axes. As another example, the intent tomove the virtual object may be determined when the user provides aninput to change a location of the virtual display via a keyboard. Theuser may push the arrow keys on the keyboard to indicate an intent tomove the virtual display to the left, right, up and/or down. As anotherexample, the user may drag the virtual display (for example, with a handgesture, with a virtual cursor, etc.) to any direction (such as left,right, up and/or down) to indicate an intent to move the virtualdisplay. As another example, the user may use a touchpad and move theuser's fingers to the left, right, up and/or down on the touchpad toindicate an intent to move the virtual display. The user may select thevirtual display when using the input devices (for example with a virtualcursor, with a hand gesture, with a voice command, etc.), withoutselecting one or more of the virtual objects. The lack of selection ofthe one or more virtual objects when using the input devices mayindicate a lack of an expression of an intent to move the virtualobject. As an even further example, a rule may define an intent of theuser. For example, if the virtual object is docked to a physical device(such as a physical input device), an intent may have already beenconveyed via the docking and a movement of the physical device.Alternatively, if no docking occurred, the intent may be conveyedthrough the absence of that docking. Intent may also be inferred by thesystem. For example, if, without a predetermined time prior to movement,a user was working with a particular object, the system might infer theuser's desire to move the object when the display moves. Thisfunctionality can be implemented through rules executed by the at leastone processor.

Some disclosed embodiments may include changing the location of thevirtual display in response to the input. A virtual display may movethrough a virtual space to change positions or locations. A change inlocation may occur when a virtual display may be moved from its originalposition or location. As stated above, the input may indicate an intentto move the virtual display from the display's current position. Theuser may indicate the intent to change location through an input device.The virtual display may be moved by any amount of distance in anydirection based on the received input. For example, the virtual displaymay be moved left, right, up and/or down. As one example, the user maypush the left arrow button on a keyboard. In response, the virtualdisplay may be moved to the left of the virtual display's originalposition. As another example, the user may move a mouse to the right. Inresponse, the virtual display may be moved to the right of the virtualdisplay's original position. As another example, the user may use atouchpad and move the user's fingers up along the touchpad. In response,the virtual display may be moved up relative to the virtual display'soriginal position.

Some disclosed embodiments may include docking the virtual display to aphysical object. A virtual object may be docked to a physical object inthe same manner as described above for docking a virtual object toanother virtual object. A physical object may include a keyboard, mouse,pen, input device, computing device, or any other object in the user'sphysical environment. By way of example, a virtual object may be dockedto a physical keyboard and moving the physical keyboard may cause thedocked virtual object to move. In some embodiments, a virtual displaymay be docked to a physical object. When the virtual display has beendocked to a physical object, movement of the physical object may cause amovement of the virtual display together with one or more virtualobjects.

After docking the virtual display to the physical object, someembodiments may further include analyzing image data captured by thewearable extended reality appliance to determine a movement of thephysical object. The wearable extended reality device may include animage sensor to capture image data. The image data may be processed todetect a movement of the physical object, for example using a visualobject tracking algorithm.

Some disclosed embodiments may include changing the locations of thevirtual display and the at least one virtual object in response to thedetermined movement of the physical object. In response to adetermination that the physical object has moved, a location of thevirtual display and one or more virtual objects may be altered. Forexample, analysis of the image data captured by the wearable extendedreality appliance, for example using a visual object tracking algorithm,may indicate that the physical object has moved in a leftward direction.In response, the virtual display and one or more virtual objects mayalso move in the leftward direction. As another example, analysis of theimage data captured by the wearable extended reality appliance mayindicate that the physical object has move in a rightward direction. Inresponse, the virtual display and the one or more virtual objects mayalso move in a rightward direction.

By way of example, FIGS. 61A and 61B illustrate an example of a virtualdisplay docked to a physical object and movements of the virtual displayin response to movements of the physical object, consistent with someembodiments of the present disclosure. As illustrated in FIG. 61A,virtual display 6110 may be located at location 6112. Physical object6111 may be located at location 6113. Physical object 6111 and virtualdisplay 6110 may be located above dashed line 6115. Virtual display 6110may be docked to physical object 6111 through docking connection 6114.As illustrated in FIG. 61B, physical object 6111 may be moved to newlocation 6119 below dashed line 6115, for example by a user. Virtualdisplay 6110 may also be moved to new location 6116, below dashed line6115, for example automatically based on docking connection 6114.

Some disclosed embodiments may include avoiding changing the locationsof the virtual display and the at least one virtual object when thedetermined movement of the physical object is smaller than a selectedthreshold. A movement of a physical object by a user may not besufficient to indicate an intent to change a location of the physicalobject and/or the virtual display. For example, in some embodiments,whether or not to move the virtual display and one or more objects maybe determined based on a distance by which the user moves a physicalobject docked to the virtual display. A threshold distance may beselected to determine when the user's intent is not to change thelocation of the physical object. By way of example, the thresholddistance may be a one or more millimeter, one or more centimeters, oneor more inches, or any other desired distance. In one example, athreshold distance may be set to 1 centimeter. In this example, when auser moves a physical object by 0.5 centimeters, a virtual displayand/or at least one virtual object docked to the physical object may notchange locations because the movement of the physical object is lessthan the threshold. As another example, when a user moves the physicalobject by 1.5 centimeters, under the same threshold requirements, thevirtual display and/or at least one virtual object docked to thephysical object may change locations because the movement of thephysical object exceeds the threshold.

In some embodiments, the threshold may be selected based on the type ofphysical object, a user's preference, a user setting, a type of movementof the physical object, the virtual display, content displayed on thevirtual display, the wearable extended reality appliance, or anotherparameter. In some embodiments, the user may set the threshold to aspecific length of time. The length of time may be 1 second, 2 seconds,5 seconds, or any other amount of time. For example, the location of thevirtual display and one or more virtual objects may be changed only whenthe movement of the physical object lasts longer than 2 seconds. Asanother example, the user may set the threshold to a specific distance.For example, the user may indicate that the location of the virtualdisplay and the at least one virtual object may change if the distancethe physical object travels is over 2 inches.

In some embodiments, docking of the virtual display to the physicalobject may take place before the docking of the at least one virtualobject to the virtual display. Multiple objects and devices may bedocked to other objects and devices. When multiple objects and devicesare docked together, all of the objects and devices may move together.At least one virtual object may be docked to the virtual display whilethe virtual display is docked to the physical object. A docking of thevirtual display to the physical object may happen first, followed by thedocking of the virtual object to the virtual display. In someembodiments, one or more virtual objects may be docked to the virtualdisplay before docking the virtual display to the physical object.

Some disclosed embodiments may include receiving input for undocking thevirtual display from the physical object, and automatically undockingthe at least one virtual object from the virtual display. An input maybe received from a user to initiate an undocking process. For example,the user may use an input device to initiate undocking. The user mayclick a button to initiate undocking. As another example, the user mayalso drag the virtual display away from the physical object in theextended reality environment to initiate undocking. In response to theinput received from the user, the virtual display may be undocked (e.g.,disconnected) from the physical object. One or more virtual objectspreviously docked to the virtual display may be automaticallydisconnected from the virtual display when the virtual display isdisconnected from the physical object, and one or more separate inputsto undock the one or more virtual objects may not be needed. Objects maynot move together as a result of the undocking. For example, the virtualdisplay may be docked to a pen and at least one virtual object, such asa phone display, may be docket to the virtual display. The user mayclick a button to initiate undocking and the virtual display may undockfrom the pen. Then the phone display may undock from the virtualdisplay. A user may move the pen, and the virtual display and the atleast one virtual object may not move as a result of the undocking.

In some embodiments, the physical object may be an input device, and theoperations may further include changing the orientations of the virtualdisplay and the at least one virtual object in response to thedetermined movement of the physical object. As described above, thephysical object may be an input device such as keyboard, mouse, pen,track ball, microphone, or other input device. The physical object maybe moved by a user and in response, the relative position of the virtualdisplay and the at least one virtual object may be adjusted. Forexample, a user may rotate a keyboard by 90 degrees. The position of thevirtual display and the at least one virtual object may be adjusted sothe virtual display and the object are also rotated by 90 degrees.

Some disclosed embodiments may include analyzing image data captured bythe wearable extended reality appliance to detect a real world event atleast partly occluded by at least the virtual display and a particularvirtual object of the plurality of virtual objects, the particularvirtual object differs than the at least one virtual object. A realworld event may include any non-virtual occurrence in a vicinity of theextended wearable reality appliance. For example, a real world event mayinclude a person and/or persons entering the space of a user. As anotherexample, a real world event may include movement of objects and/orpeople towards or away from the user. As another example, a real worldevent may include movement of the user towards or away from people. Thewearable extended reality appliance, or an external computing device,may detect a real world event based on an analysis of the image datareceived from an image sensor included in the wearable extended realityappliance, for example using a visual event detection algorithm. Forexample, the wearable extended reality appliance or the externalcomputing device may detect a person walking, people organizing in agroup, or any other event taking place around the user. The real worldevent may be partially or fully occluded from view of the user by thepresentation of the virtual display and/or the particular virtual objectvia the wearable extended reality appliance. For example, a person mayenter behind a virtual display, and the user may not be able to see theperson because the virtual display may block the user's view. As anotherexample, an object may be moved so that the object is behind a virtualobject. The user may not be able to see the object because the virtualobject may block the user's view. In order for the user to see the realworld event, the virtual display and the particular virtual object maybe moved.

Some embodiments may include, in response to the detection of the realworld event at least partly occluded by at least the virtual display andthe particular virtual object, moving the virtual display and the atleast one virtual object in a first direction, and moving the particularvirtual object in a second direction, the second direction may differfrom the first direction. The virtual display and the at least onevirtual object may move in a direction that is away from the real worldevent. The virtual display, at least one virtual object, and aparticular object may move in the same direction or in differentdirections, depending on, for example, an amount of space available inthe extended reality environment for displaying the virtual display, oneor more virtual objects, and/or the particular object at one or more newlocations. For example, a virtual display, at least one virtual object,and a particular object may all move to the left to prevent occlusion ofa real world event. As another example, the particular virtual objectmay move in a second direction that is also away from the real worldevent. In another example, the particular object may move in a differentdirection than the virtual display and the virtual objects. For example,the real world event may be obscured by the virtual display and thevirtual objects. The virtual display and the at least one virtual objectmay move in a rightward direction and the particular virtual object maymove in a leftward direction to prevent occlusion of the real worldevent. By way of another example, information on the virtual display mayreorient to locations other than those overlapping with the real worldoccurrence so as to permit viewing of the real world occurrence.

Some disclosed embodiments may include displaying text entered using theinput device in the virtual display. The input device may include aninterface that allows a user to create text. A user may create text bytyping on a keyboard, using a stylus, writing on a touchpad, usingspeech to text conversion or any other way of creating text. The textcreated using the input device may appear on the virtual display. Forexample, the input device may be a keyboard. The user may push a “K” keyon the keyboard and the text, “K”, may appear on the virtual display. Asanother example, the input device may be a microphone. The user mayspeak into the microphone and say “Hello” and the text, “Hello” mayappear on the virtual display.

In some embodiments, changing the location of the virtual display, maycause the at least one virtual object to move with the virtual displayas a result of the docking of the at least one virtual object to thevirtual display. The location of the virtual display may be moved anydistance in any direction. The docked virtual object may move the samedistance and/or a proportional distance in the same direction as thevirtual display. For example, an input may be received from the userindicative of an intent to move the virtual display to the right. Thevirtual display may move to the right of the virtual display's originalposition and the virtual object may also move to the right of thevirtual object's original position by the same amount or by a differentamount. In another example, the at least one virtual object may be movedto a position that preserves a spatial relation (such as direction,distance, orientation, and so forth) between the at least one virtualobject and the virtual display. In some embodiments, a physical objectmay be detected in a particular area of the environment. For example,the physical object may be detected by analyzing image data capturedusing an image sensor, for example using an object detection algorithm.The image sensor may be included in the wearable extended realityappliance, in an input device, in a computing device, and so forth. Inanother example, the physical object may be detected using radar, Lidar,sonar, and so forth. Further, to preserve the spatial relation betweenthe at least one virtual object and the virtual display, the at leastone virtual object may need to be moved to the particular area. Inresponse to the detection of the physical object in the particular area,the movement of the at least one virtual object to the particular areamay be avoided. In one example, the movement of the at least one virtualobject in response to the change in the location of the virtual displaymay be avoided, and the at least one virtual object may be left in itsoriginal position. In another example, the at least one virtual objectmay be moved to an alternative area of the environment in response tothe change in the location of the virtual display and the detection ofthe physical object in the particular area.

Some disclosed embodiments may include causing the at least one virtualobject to move from a first location to a second location, wherein aspatial orientation of the at least one virtual object relative to thevirtual display in the second location corresponds to an originalspatial orientation of the at least one virtual object relative to thevirtual display in the first location. Spatial orientation may refer torelative positions. For example, a position of a first object relativeto a second object may refer to one or more distances and/or directionsof the first object relative to the second object. Spatial orientationmay be measured by distances, rotational angles, or both. Spatialorientation may be determined in a two-dimensional or athree-dimensional space. As discussed above, a virtual object may movewhen it is docked to a virtual display and the virtual display moves.After the virtual object and virtual display have moved from their firstlocation, the space between the virtual object and the virtual displaymay remain the same at the second location. The virtual object mayoccupy the same position and/or angular orientation relative to thevirtual display. For example, the virtual display may be moved to theleft of the first location. The virtual object may also be moved to theleft of the first location. The virtual object may stay the samedistance from the virtual display in the second location as in the firstlocation. As another example, the virtual object may be oriented at anangle of 90 degrees relative to the virtual display. The virtual displaymay be turned to the right direction. The virtual object may also beturned so that the object may still be oriented at an angle of 90degrees relative to the virtual display.

Some disclosed embodiments may include receiving a selection of anadditional virtual object from the plurality of virtual objects. Inaddition to the first virtual object, the user may select another (e.g.,a second) virtual object using an input device as previously described.Other disclosed embodiments may also include docking the additionalvirtual object to the at least one virtual object. As discussed above,virtual objects may be docked to virtual displays or to other virtualobjects. The second or additional virtual object may be docked to thefirst virtual object and/or to the virtual display in a manner similarto the docking described earlier. After docking the additional virtualobject to the at least one virtual object, additional embodiments mayinclude receiving a second input, the second input is indicative of asecond intent to change the location of the virtual display without anexpression of an second intent to move either the at least one virtualobject or the additional virtual object. As stated above, the user mayuse an input device to create an intent to move the virtual displaywithout attempting to move the additional virtual object. For example, auser may use a mouse to drag and move the virtual display but may notmove the additional virtual object.

Some disclosed embodiments may include changing the location of thevirtual display in response to the second input. The second input may beone of the types of inputs discussed earlier with regard to the firstinput.

In some embodiments, changing the location of the virtual display, maycause the at least one virtual object and the additional virtual objectto move with the virtual display as a result of the docking of the atleast one virtual object to the virtual display and the docking of theadditional virtual object to the at least one virtual object. The atleast one virtual object and the additional virtual object may move inresponse to the change in location of the virtual display. For example,at least one virtual object may be docked to a virtual display and anadditional virtual object may be docked to the at least one virtualobject. A user may use a mouse to drag and move the virtual display inan upward direction. The first virtual object and the additional virtualobject may also be moved in an upward direction since the virtualobjects are docked to the virtual display. The location of the virtualdisplay may be moved any distance in any direction. The docked at leastone virtual object and the additional virtual object may move the samedistance and/or a proportional distance in the same direction as thevirtual display. For example, an input may be received from the userindicative of an intent to move the virtual display to the right. Thevirtual display may move to the right of the virtual display's originalposition. The at least one virtual object and the additional virtualobject may also move to the right of the at least one virtual object'soriginal position and the additional virtual object's original positionby the same amount or by a different amount.

By way of example, FIGS. 62A and 62B illustrate an example of a virtualdisplay and a plurality of virtual objects and illustrate changes inlocations of one or more of the virtual objects when the virtual displaychanges locations, consistent with some embodiments of the presentdisclosure. As illustrated in FIG. 62A, virtual display 6210 may belocated at location 6213. Virtual object 6211 may be located at location6214 and may be docked to virtual display 6210, as illustrated by thesolid line connecting the two. Virtual object 6212 may be located atlocation 6215 and may be docked to virtual object 6211, as illustratedby the solid line connecting the two. It is understood that these solidlines may not actually exist, but are only used to illustrate thedocking. Such solid lines, or other visual indications of docking, mayor may not be presented in the extended reality environment. Virtualobjects 6211 and 6212 and virtual display 6210 may be located abovedashed line 6216. As illustrated in FIG. 62B, virtual display 6210 maybe moved to new location 6218, below dashed line 6216. Virtual object86211 may also move to new location 6220 below dashed line 6216, basedon the docking between virtual object 6211 and virtual display 6210.Similarly, virtual object 6212 may also move to new location 6222 belowdashed line 6216, based on the docking between virtual object 6211 andvirtual object 6212.

In some embodiments, selectively moving the at least one virtual objectwith the virtual display may be geolocation-dependent. The location ofthe user may impact how the system responds. Responses may differ forexample when the user is in a private space such as an office, verses apublic space. Or a response may differ based on whether the user is in aprivate office versus a conference room. Thus, the at least one virtualobject may move with the virtual display based on a geographicallocation. The geographical location may be based on the GPS coordinatesof the user or other location-based detection such as indoor locatingtechnology based on WiFi or other signals. GPS coordinates may uselatitude and longitude to determine the location of the user, thewearable extended reality appliance, or any other component of thesystem. The movement may be geolocation-dependent for privacy orsecurity reasons. For example, the user may be in the home of the user.The at least one virtual object may move with the virtual displaybecause the user is at home. As another example, the wearable extendedreality appliance may be in the office of the user. The at least onevirtual object may move with the virtual display because the wearableextended reality appliance is in a user's office. As another example,the user may be in a public space. The at least one virtual object maynot move with the virtual display because the user is in a publicsetting and the user may not want the virtual objects to be viewable inpublic.

Some disclosed embodiments may include causing the at least one virtualobject to move with the virtual display upon detection of the wearableextended reality appliance at a first geolocation and disassociating theat least one virtual object from the virtual display upon detection ofthe wearable extended reality appliance at a second geolocation,different from the first geolocation. When the wearable extended realityappliance is at a first geographical location, the at least one virtualobject and the virtual display may move together. When the wearableextended reality appliance is at a second geographical location, the atleast one virtual object and the virtual display may not move together.The first geographical location may be in a private setting, while thesecond geolocation may be in a public setting. For example, the wearableextended reality appliance may be located in the office of the user.When the wearable extended reality appliance is in the office of theuser the at least one virtual object may move in the same direction asthe virtual display. For example, the user may use a mouse to drag thevirtual display in a right direction. The at least one virtual objectmay also be moved in a right direction. As another example, the wearableextended reality appliance may be located in a public space. While inthe public space, the at least one virtual object may not be associatedwith the virtual display. Therefore, if a user moves a virtual display,the at least one virtual object may not move in the same direction asthe virtual display. For example, if the user moves the virtual displayin a downward direction, the at least one virtual object will stay inthe same position.

After docking the at least one virtual object to the virtual display,some embodiments may include, receiving a first user-initiated input fortriggering a change in the location of the virtual display and fortriggering a change in the location of the at least one virtual object,receiving a second user-initiated input for triggering a change in thelocation of the virtual display, wherein the second user-initiated inputexcludes a trigger for a change in the location of the at least onevirtual object, changing the location of the virtual display and the atleast one virtual object in response to the first user-initiated input,and changing the location of the virtual display and the at least onevirtual object in response to the second user-initiated input. A usermay provide a first user-initiated input that is directed to changingthe position of both the virtual display and the at least one virtualobject. For example, the user may select both the virtual display andthe at least one virtual object with a mouse, and then drag both withthe mouse to change positions. The user may also provide a seconduser-initiated input that is directed only to the virtual display. Thelocation of both the virtual display and the at least one virtual objectmay be changed based on the first and second user-initiated input. Forexample, the user may use an input device to provide a user-initiatedinput. The input device may be a keyboard, mouse, touchpad, or otherdevice. The user may use the input device to trigger a change inlocation by using the device to move the virtual display. For example,the user may provide a first input by using a mouse to click and dragthe virtual display and the virtual object. The location of the virtualdisplay and the virtual object may change based on the location of wherethe mouse drags the virtual display and the virtual object. For example,the mouse may drag the virtual display and the virtual object in theleft direction, and the virtual display and the virtual object may movein the left direction. As another example, the mouse may drag thevirtual display and the virtual object in an upward direction, and thevirtual display and the virtual object may move in the upward direction.As another example, a user may use a touchpad to create a seconduser-initiated input by clicking and dragging only the virtual displayin a downward direction. The virtual object may also move in a downwarddirection since the virtual object may be docked to the virtual display,even though the second user-initiated input may not be directed to thevirtual object.

By way of example, FIGS. 62A and 62B illustrate a virtual display and avirtual object moving as a result of a second-user initiated input.Virtual display 6210 may move from location 6213 in FIG. 62A to location6218 in FIG. 62B, based on a user clicking and dragging virtual display6210 using a mouse. The user may be excluding a change in location ofvirtual objects 6211 and 6212 by not clicking and dragging virtualobjects 6211 and 6212. Virtual objects 6211 and 6212 may move to newlocations 6220 and 6222 in FIG. 62B because virtual objects 6211 and6212 are docked with virtual display 6210.

Some disclosed embodiments may include receiving a third user-initiatedinput that triggers a change in the location of the at least one virtualobject but excludes change in the location of the virtual display, andchanging the location of the virtual display and the at least onevirtual object in response to the third user-initiated input. A user mayprovide a third user-initiated input that is directed to changing theposition of only the at least one virtual object. The user-initiatedinput, as described above, may create a change in the location of thevirtual display and the at least one virtual object when the virtualdisplay and the virtual object are docked together. The thirduser-initiated input may be of one or more of the types of inputdescribed earlier in connection with the first user-initiated input. Forexample, a mouse may drag the at least one virtual object in a downwarddirection, and the at least one virtual object may be moved in adownward direction. The virtual display may also move in the samedownward direction as the at least one virtual object.

By way of example, FIGS. 60A and 60B illustrate a virtual display movingas a result of a third user-initiated input. Virtual object 5912 maymove from location 5916 to location 6022 in FIG. 60B based on a userclicking and dragging virtual object 5912 using a mouse. The user may beexcluding a change in location of virtual display 5910 by not clickingand dragging virtual display 5910. Virtual display 5910 and virtualobject 5911 may move to new locations 5926 and 6024 in FIG. 60B becausevirtual display 5910 and virtual object 5911 are docked to virtualobject 5912.

Some disclosed embodiments may include displaying the virtual display ona first virtual surface and displaying the at least one virtual objecton a second surface that at least partially coincides with the firstsurface. A virtual surface may be a surface that exists in an extendedreality environment. The virtual surface may have any shape. Forexample, the virtual surface may have a square, rectangular, circular,or other shape. The virtual surface may also not have a defined shape.There may be a single virtual surface or multiple virtual surfaces. Thevirtual display and the at least one virtual object may be projectedonto different virtual surfaces. The virtual display may be projectedonto a first virtual surface and the at least one virtual object may beprojected onto a second virtual surface. The first and second virtualsurfaces may have portions that partially or fully overlap, touch, orintersect with each other. For example, the first virtual surface may bea touchpad and the second virtual surface may be a keyboard. Thetouchpad may have an edge that touches an edge of the keyboard.

FIG. 63 illustrates a flow chart of an exemplary method that may beexecuted by a processor to perform operations for docking virtualobjects to virtual display screens in an extended reality environment.Method 6300 may include a step 6310 of generating virtual content forpresentation via a wearable extended reality appliance, where thevirtual content includes a virtual display and a plurality of virtualobjects located outside the virtual display. Method 6300 may alsoinclude a step 6311 of receiving a selection of at least one virtualobject from the plurality of virtual objects. Method 6300 may alsoinclude a step 6312 of docking the at least one virtual object to thevirtual display. Further, method 6300 may include a step 6313 ofreceiving an input indicative of an intent to change a location of thevirtual display. Method 6300 may also include a step 6314 of changingthe location of the virtual display in response to the input, andwherein changing the location of the virtual display causes the at leastone virtual object to move with the virtual display.

A virtual working environment may include a plurality of virtual objectsand physical objects. Physical objects may include articles such askeyboards, computer mice, trackpads, physical screens, or any otherphysical object. Virtual objects may include items such as virtualscreens, or virtual objects displayed on a virtual screen. A virtualobject may, for example, virtually represent something physical.Avatars, icons, simulations, graphical representations are just a fewexamples of items that may be virtual objects (virtual items). Thevirtual objects and the physical objects may be associated with virtualplanes. That is, some types of objects may be depicted in alignment witha virtual plane. Theoretically, virtual planes may extend infinitely inany direction. A set of virtual objects and a physical object may all belocated in a first plane and may be docked to each other. Docking mayinvolve a form of linking whereby movement of an object docked toanother may cause that other object to move. Thus, for example, movementof a physical object may trigger movement of the associated docked setof virtual objects in the first plane. Alternatively, or additionally, aset of virtual objects may be docked to a physical object (e.g., akeyboard). The set of virtual objects and the physical object may belocated on different virtual planes. Movement of the physical object maytrigger movement of the associated docked set of virtual objects,regardless of whether they are located in a common plane.

Some disclosed embodiments may involve implementing selective virtualobject display changes. Display changes may include changing howsomething may be visually presented to a user. For example, displaychanges may include changing size, moving location, orientation,adjusting brightness and/or color, changing resolution, and/or any othervisually perceptible alterations. The display changes may be applied tosome but not all virtual objects. Virtual objects may exist in anextended reality environment. A virtual object may appear when a userinteracts with that extended reality environment. For example, a virtualobject may include virtual display screens, widgets, documents,presentations, media items, photos, videos, virtual characters, scrollbars, adjusters, and other objects. A user may be able to view, move,rotate, flip, squeeze, enlarge, modify, and/or interact with virtualobjects, in order to achieve functionalities, such as altering,examining or moving the virtual objects. A visual presentation of avirtual object may be changed. For example, a user may adjust the sizeof a virtual object. As another example, a user may rearrange thelocations of one or more virtual objects. A user may be selective inexecuting virtual object display changes, for example by choosing tochange some virtual objects but not others, by selecting a type oranother parameter of a change, and so forth. For example, a user maychoose to move one virtual object but keep the location of anothervirtual object the same. As another example, a user may choose to adjustthe brightness of one virtual object but not adjust the brightness of adifferent virtual object. In yet another example, a user may select adirection of movement or a new location for a virtual object when movingthe virtual object.

Some disclosed embodiments may involve generating an extended realityenvironment via a wearable extended reality appliance, the generatedextended reality environment may include a first virtual planeassociated with a physical object and a second virtual plane associatedwith an item, the second virtual plane may extend in a directiontransverse to the first virtual plane. In one example, generating theextended reality environment may comprise presenting at least part ofthe extended reality environment to a user of the wearable extendedreality appliance. A virtual plane may be a dimensional surface that maybe used to represent the surface of an article, and which may extendbeyond boundaries of that article, depending on design choice. Multiplevirtual planes (for example, two, three, more than three, more than ten,infinity, or any other number of virtual planes) may exist and mayextend in multiple different directions. A virtual plane may beconsidered virtual if it is employed in connection with an extendedreality display or an extended reality environment, regardless ofwhether the plane is visible. Specifically, a virtual plane may bedisplayed in color or with texture so that it may be visible to a wearerof an extended reality appliance, or the virtual plane may be invisibleto the eye, but might become perceptible when visible objects arelocated in the virtual plane. In one example, a virtual plane may beillustrated with virtual grid lines in an extended reality environment.A virtual plane may include, for example, a flat surface, a non-flatsurface, a curved surface, or a surface having any other desiredconfiguration. For example, a virtual plane may be a flattwo-dimensional surface, a curved two-dimensional surface, an eventwo-dimensional surface, a homogenous two-dimensional surface, anon-homogenous two-dimensional surface, and so forth. A physical objectand an item may exist in the same virtual plane or in separate virtualplanes. For example, a first virtual plane may be a virtual plane of thephysical object (e.g., a top surface of a table or desk) and the virtualplane may extend horizontally. In another example, a first virtual planemay be a virtual plane associated with a physical surface that thephysical object is placed on (e.g., a keyboard placed on a top surfaceof a table or desk) and the virtual plane may extend to one or moredirections. As another example, a second virtual plane may be a virtualplane of the item (e.g., a virtual display screen, a virtual controlplane, a physical item, etc.) and the virtual plane may extendvertically.

Virtual planes that exist together in an extended reality environmentmay extend across each other. That is, virtual planes may intersect eachother. Virtual planes in an extended reality environment may also extendin the same direction and never cross each other. Virtual planes mayalso extend across (or intersect) with two or more other virtual planesat the same or different angles. For example, a second virtual plane maytransverse a first virtual plane by extending across the first virtualplane at a right angle. As another example, a first virtual plane mayextend across a second virtual plane at a 75° angle. As another example,a first virtual plane may extend in a horizontal direction and a secondvirtual plane may also extend in a horizontal direction, so the twoplanes never extend across each other.

By way of example, FIG. 64 illustrates an example of a physical objectin a first plane and an item in a second plane, consistent with someembodiments of the present disclosure. In some embodiments, firstvirtual plane 6410 may exist in an extended reality space. Physicalobject 6411 may be associated with first virtual plane 6410. Secondvirtual plane 6420 may also exist in the extended reality space andextend in a direction transverse to first virtual plane 6410. Item 6421,which may be virtual or physical, may be associated with second virtualplane 6420.

In some embodiments, the first virtual plane may be flat and the secondvirtual plane may be curved. In another example, the first virtual planemay be flat and the second virtual plane may be flat. In yet anotherexample, the first virtual plane may be curved and the second virtualplane may be flat. In another example, the first virtual plane may becurved and the second virtual plane may be curved. A flat virtual planemay be a flat surface in which a straight line between any two points onthe virtual plane lies wholly within or on that surface. For example, afirst virtual plane may extend along a horizontal plane and movement ofa first set of virtual objects in the first virtual plane may occur in atwo-dimensional space. A curved plane may be a curved surface in which astraight line between at least one pair of two points on the virtualplane lies outside the surface. A curved plane may extend in athree-dimensional space. For example, a second virtual plane may extendalong a curved path and movement of a first set of virtual objects inthe first virtual plane may occur in a three-dimensional space.

In some embodiments, the physical object may be located on a physicalsurface with the first virtual plane extending beyond dimensions of thephysical surface. A physical surface may be a surface of a physicalobject present in the user's environment. For example, a physicalsurface may be a desk, table, chair, dresser, couch, bookshelf,countertop, or any other surface. A physical object may be positioned ona physical surface. A first virtual plane may represent the space aphysical object occupies and may extend past the boundaries of aphysical surface. As another example, a physical object may be locatedon a desk in a room and a virtual plane may coincide with an uppersurface of the desk and extend past the end of the desk to the walls ofthe room. By way of another example, a physical object may be located ona table in an office and a virtual plane may coincide with an uppersurface of the table and extend past the end of the table to a window inthe room.

Some disclosed embodiments may involve accessing a first instruction fordocking a first set of virtual objects in first positions associatedwith the first virtual plane. Docking, in this context, may includeconnecting, tethering, linking or otherwise joining a virtual object toa particular position on a virtual plane. Docking may ensure that theobjects stay in the particular position on the virtual plane when thevirtual plane moves. In one example, instructions may be a set ofcommands or rules for performing a task or an action. In anotherexample, instructions may include any information configured to cause adesired action. Instructions may be provided to a processor to executethe instructions to perform the task—in this case linking one or moreobjects to particular locations, which, in some embodiments may bedefined by coordinates. Accessing an instruction, for example by theprocessor, may include at least one of: receiving the instruction,reading the instructions from memory, receiving the instruction from anexternal device, determining the instruction by analyzing data,receiving the instruction from a user (for example, through a userinterface, through gestures, through voice commands, etc.), and soforth. For example, an instruction may be to associate a coordinate on avirtual plane with a particular object, for example in a data-structure.Instructions may define the rules for how the docking should take place.Thus, a first set of instructions may define that a virtual object maybe docked to a virtual plane in a specific position. The virtual planemay move, and the docked virtual object may also move with the virtualplane, to ensure that the virtual object stays in the same position onthe virtual plane. One or more virtual objects (for example, one virtualobject, at least two virtual objects, at least three virtual objects, atleast ten virtual objects, etc.) may make up a set of virtual objects. Aset of virtual objects may be docked to the same position on a virtualplane, or each object may be docked to a different position. Forexample, a set of virtual objects may be docked in a position that maybe three inches to the left of a physical object and on a virtual planeassociated with the physical object. As another example, a set ofvirtual objects may be docked in a position that may be two centimetersup from a physical object and on a virtual plane associated with thephysical object. In some embodiments, one or more objects in the set ofvirtual objects may be docked to different positions on the virtualplane. Thus for example, a first virtual object may be docked at acoordinate to the left of a physical object and a second virtual objectmay be docked at a coordinate to the right of the physical object.

Some disclosed embodiments may involve accessing a second instructionfor docking a second set of virtual objects in second positionsassociated the second virtual plane. The second instruction may havecharacteristics similar to those described above for the firstinstruction. A second set of virtual objects, like the first set ofvirtual objects, may include one or more virtual objects. A second setof virtual objects may be docked to the same position or to differentpositions in the same manner as described above with respect to thefirst set of virtual objects. For example, a second set of virtualobjects may be docked in a position that may be three inches to the leftof an item and on a virtual plane associated with the item. As anotherexample, a second set of virtual objects may be docked in a positionthat may be two centimeters up from an item and on a virtual planeassociated with the item.

Some disclosed embodiments may involve receiving a first inputassociated with a movement of the physical object. Movement of aphysical object may include changing locations or orientations within aphysical space. For example, movement may include a change in distance,position, elevation, angular orientation, or any other type of movement.A physical object may move as a result of a user's actions. For example,a user may move a physical object five inches to the right on a table.In some examples, receiving the first input may include at least one ofreading the first input from memory, receiving the first input from anexternal device, or capturing the first input using a sensor. Themovement of the physical object may be detected by an image sensor andwhich constitute an input that may be received by one or moreprocessors. In other words, signals from an image sensor may beanalyzed, for example using visual object tracking algorithm, todetermine that the physical object was moved in a certain direction by acertain distance, and/or that the physical object was rotated in acertain direction.

In some embodiments, the physical object may be a computing device andthe first input may include motion data received from at least onemotion sensor associated with the computing device. A computing devicemay be an electronic device controlled by a digital processing unit. Forexample, a computing device may include a smart phone, a laptopcomputer, a tablet, a desktop computer, a hand-held computer, a wearablecomputer, or any other type of computing device. The computing devicemay contain a motion sensor. A motion sensor may be an electronic devicedesigned to detect and measure linear and/or angular movement. Movementof a computing device as detected by the motion sensor may be collectedas motion data, and the motion data may be received as the first input.For example, a smart phone may be moved to the left in a physical space.A motion sensor may store the leftward movement of the smart phone as afirst input. A motion sensor may be located anywhere on a computingdevice. For example, a motion sensor may be located on a side of alaptop computer. As another example, a motion sensor may be located on atop portion of a laptop computer. As another example, a motion sensormay be located on a bottom portion of a laptop computer.

Some disclosed embodiments may involve analyzing the motion data todetermine if the movement of the physical object is greater than athreshold, causing the change in the display of the first set of virtualobjects when the movement of the physical object is greater than thethreshold, and maintaining the display of the first set of virtualobjects when the movement of the physical object is less than thethreshold. A threshold may be the magnitude or intensity of a parameterthat must be exceeded for a certain reaction. The threshold may be basedon distance, speed, rotation angle, acceleration, force, or any otherparameter. The threshold may be pre-determined, may be set by a user, ormay be determined based on current state of the system or of theenvironment. For example, the threshold may be based on a type of thephysical object, on a size of the physical object, on a distance betweenthe physical object and the second virtual plane, on a type of aphysical surface that the physical object is placed on, on a size of aphysical surface that the physical object is place on, on a distancebetween the physical object and a user, and so forth. Motion data may beanalyzed to detect motion and determine if the movement is greater than,equal to, or smaller than the threshold. In some cases, a movement of aphysical object may be relatively small (e.g., less than the threshold)and as a result there may be no change in the display of a first set ofvirtual objects. That is, the first set of virtual objects may bemaintained in their respective positions on the virtual plane.Alternatively, movement of a physical object may be relatively large(e.g., greater than or equal to the threshold) and as a result thedisplay of a first set of virtual objects may change. That is, the firstset of virtual objects may be moved from their positions in response tothe movement of the physical object. For example, when a threshold isset to five centimeters and a physical object is moved by only threecentimeters, a display of a first set of virtual objects may not bechanged because the distance by which the physical object is moved islower than the threshold. As another example, if a user in this scenariowere to move the physical object eight centimeters, a display of a firstset of virtual objects may be changed because the distance by which thephysical object moved exceeds the threshold.

In some embodiments, the physical object is an inanimate object and thefirst input may include image data received from an image sensorassociated with the wearable extended reality appliance. An inanimateobject may be an object that is unable to move on its own. For example,an inanimate may be a keyboard, a chair, a writing utensil, a notebook,or any other object not capable of changing its position or orientationwithout being subjected to some external stimulus. A wearable extendedreality appliance may contain an image sensor to store images of thephysical object. An image sensor may be an electronic device configuredto capture an optical image and convert that optical image into anelectronic signal. An image sensor may be attached to any part of awearable extended reality appliance, as described above. Images and/orvideos of a physical object may be captured by the image sensor andprovided as image data. In some embodiments, the first input may includeimage data of the physical object captured by the image sensor.

Some disclosed embodiments may involve analyzing the image data todetermine if the movement of the physical object was prompted by a userof the wearable extended reality appliance, causing the change in thedisplay of the first set of virtual objects when the movement of thephysical object was prompted by the user, and maintaining the display ofthe first set of virtual objects when the movement of the physicalobject was not prompted by the user. A user of the wearable extendedreality appliance may prompt movement of a physical object by physicallypushing and/or pulling the physical object and changing the physicalobject's location, position, orientation, or any other by moving thephysical object in any other way. Alternatively, a person, other thanthe user, may also prompt movement of a physical object in the samemanner. An image sensor may capture images of a person moving thephysical object. The images may be analyzed to determine if the personwho moved the physical object is a user of a wearable extended realityappliance. In one example, this may occur by analyzing physical humanbody characteristics of the hands or other associated body parts of themover captured by an image sensor, and comparing those characteristicswith stored characteristics of the user. One such mechanism may befacial or hand recognition. Facial or hand recognition technology mayuse known computer algorithms to pick specific, distinctive details of auser's face or hands, which may be compared to stored details of otherfaces and/or hands that have been collected in a database. In anotherexample, a binary visual classification algorithm may be used to analyzethe images of a person moving the physical object and classy it to oneof two classes, ‘movement of the physical object was prompted by theuser of the wearable extended reality appliance’ or ‘movement of thephysical object was not prompted by the user of the wearable extendedreality appliance’. Such binary visual classification algorithm may beobtained by training a machine learning model using training examples.In one example, a user of a wearable extended reality appliance may movea physical object. Image data obtained by the image sensor may beanalyzed to determine whether the user moved the physical object. Whenit is determined that the physical object was moved by the user, adisplay of a first set of virtual objects may be changed based on theuser's actions. That is, the first set of virtual objects may also bemoved based on the user's actions. As another example, an individual notwearing the wearable extended reality appliance may move a physicalobject, or the physical object may otherwise move not due an action ofthe user wearing the wearable extended reality appliance. Image dataobtained by the image sensor may be analyzed to determine whether theuser moved the physical object. When it is determined that the physicalobject was moved by someone other than the user (or that the physicalobject was not moved by the user), a display of a first set of virtualobjects may not be changed. That is, the first set of virtual objectsmay not be moved based on the movement of the physical object by someoneother than the user of the wearable extended reality appliance (or basedon the movement of the physical object not being by the user of thewearable extended reality appliance).

In response to receiving the first input, some disclosed embodiments mayinvolve, causing a change in a display of the first set of virtualobjects in a manner corresponding the movement of the physical objectwhile maintaining the second set of virtual objects in the secondpositions. Causing a change in a display may include changing how thevirtual objects are visually presented to a user, for example via thewearable extended reality appliance. For example, a change in displaymay include a change in location, a change in visual properties, such assize, brightness, saturation, orientation, opacity, intensity, graphicaltheme, color scheme, and/or other display changes. A first set ofvirtual objects may be located on a same virtual plane as a physicalobject and a second set of virtual objects may be located on a differentvirtual plane. The physical object may change locations on the virtualplane and create an input (e.g., via signals from a motion sensor or animage sensor). A display of a first set of virtual objects may bechanged based on the input. A display of a second set of virtual objectsmay not be changed. For example, a physical object may move three feetto the right in the first virtual plane. A first set of virtual objectsmay also move three feet to the right in the first virtual plane, whilethe second set of virtual objects may not move in the second virtualplane.

By way of example, FIG. 65 illustrates an example of virtual objectsdocked to positions in a virtual plane prior to movement of the physicalobject, consistent with some embodiments of the present disclosure. Insome embodiments, physical object 6511 and first set of virtual objects6512 may be associated with first virtual plane 6510. First set ofvirtual objects 6512 may be docked to positions on first virtual plane6510. Item 6521 and second set of virtual objects 6522 and 6523 may beassociated with second virtual plane 6520. Second set of virtual objects6522 and 6523 may be docked to positions on second virtual plane 6520.

By way of example, FIG. 66 illustrates an example of a movement of aphysical object and virtual objects relative to the situationillustrated in FIG. 65, consistent with some embodiments of the presentdisclosure. In some embodiments, physical object 6611 and first set ofvirtual objects 6612 may be associated with first virtual plane 6610.First set of virtual objects 6612 may be docked to positions on firstvirtual plane 6610. Item 6621 and second set of virtual objects 6622 and6623 may be associated with second virtual plane 6620. Second set ofvirtual objects 6622 and 6623 may be docked to positions on secondvirtual plane 6620. Physical object 6611 may be moved from its originalposition (shown as 6515) to a new position (shown as 6616). First set ofvirtual objects 6612 (may be moved to correspond with the movement ofphysical object 6611. However, second set of virtual objects 6622 and6623 may not move.

In some embodiments, when the physical object is moved to a new locationdetermined to be separated from the physical surface on which thephysical object was originally located, the display of the first set ofvirtual objects may be updated to appear in proximity to the newlocation but with a modified display of the virtual objects. Forexample, if an original location of a physical object was a surface of aconference room table in a first plane, and the new location is asurface of a side table in a second plane, the virtual objects may bothmove to an associated location with the side table (e.g., a movementproportional to the movement of the physical object), and also change inappearance. For example, the change in planes may necessitate aperspective change to be reflected in the virtual objects. In anotherexample, the new position of the virtual objects in differentillumination conditions may necessitate an opacity or a brightnesschange. In yet another example, partial occlusions of the virtualobjects in the extended reality environments may necessitate a partialrendering of the virtual objects. Thus, not only will the virtualobjects move to a new location, but their appearances in the newlocation may differ from their prior appearances. In some embodiments, anew location may no longer be on a physical surface. A display of thevirtual objects may be changed even if the new location is no longer onthe same physical surface. For example, a physical object may be locatedon a desk before being moved. After being moved, the physical object maybe located on a chair. A first set of virtual objects may also be movedto be located in an associative manner with the chair.

In some embodiments, modifying the display of the first set of virtualobjects includes at least one of causing the first set of virtualobjects to disappear, changing at least one visual property of the firstset of virtual objects, or displaying a minimized version of the firstset of virtual objects. Modifying a display may include implementing achange in visual property, appearance, or other visual characteristicsassociated with the display. A visual property may be the characteristicappearance of an object. Visual characteristics may include size,orientation, opacity, intensity, graphical theme, color scheme,alignment, spacing, and/or other parameters that may control thedisplay. For example, a display may be modified by completing removing(e.g., deleting) a first set of virtual objects from an extended realityenvironment. As another example, a size of a first set of virtualobjects may be decreased. As another example, a brightness of a firstset of virtual objects may be adjusted.

Some disclosed embodiments may involve receiving a second inputassociated with a movement of the item. In some examples, receiving thesecond input may include at least one of reading the second input frommemory, receiving the second input from an external device, or capturingthe second input using a sensor. An item may be moved in the same manneras a physical object described above. For example, an item (such as aphysical item, a virtual item, an icon, an avatar or a widget) may bemoved three inches in an upward direction. Similar to a first input, amovement of the item may constitute a second input that may be receivedby one or more processors. In one example, the item may be a virtualitem in an extended reality environment, and the second input may bereceived from a wearable extended reality appliance presenting the item,from a computerized system coordinating the extended realityenvironment, from a computerized system controlling the presentation ofthe item via one or more wearable extended reality appliances, and soforth.

In some embodiments, the item is a virtual object and the second inputincludes pointing data received from an input device connectable to thewearable extended reality appliance. Pointing data may include data usedto identify a location. Such data may include, for example, coordinatelocations, distances and/or angular positions relative to a particularreference axis or plane. A user may identify a point using an inputdevice. An input device may be configured to allow one or more users toinput information. For example, a user may identify a point (e.g., alocation) by touching a touchpad, controlling a keyboard, moving acomputer mouse, using a touchscreen, through hand gestures, throughvoice commands, or by any other means. For example, an item may be avirtual screen. A user may drag the virtual screen with a virtual cursorcontrolled by a computer mouse, may push or pull the virtual screen withhand gestures, etc., and in response a second input that may be receivedby one or more processors associated with some disclosed embodiments.

Some disclosed embodiments may involve analyzing the pointing data toidentify cursor actions indicative of a desired movement of the virtualobject, and causing the change in the display of the second set ofvirtual objects in a manner corresponding with the desired movement ofthe virtual object. A cursor may be a movable indicator that identifiesa point. Cursor actions may involve a user controlling a cursor toidentify a point by hovering over the point, clicking on the point,dragging the point, or any other action that may cause the appearanceand/or position of the point to be altered. Cursor actions may beanalyzed to determine that the user may wish to select and move avirtual object. In response, a display of a second set of virtualobjects may be changed relative to a movement of a virtual object by auser. For example, an item may be a virtual screen and a second set ofvirtual objects may be widgets. A user may move the virtual screen bydragging the screen using a cursor. In response, a second set of virtualobjects may also move in the direction of movement of the screen. Amovement of the second set of virtual objects may be in proportion to amovement of the item. For example, an item may be moved five inches, anda second set of virtual objects may be moved five inches.

In some embodiments, the item may be a virtual object and the secondinput may include image data received from an image sensor associatedwith the wearable extended reality appliance. Image data associated witha virtual object may be captured by an image sensor in the same manneras described above with respect to the image data of the physicalobject. An image sensor may capture an image or a video, and the secondinput may include the image or video, or may be based on an analysis ofthe image or video. For example, the item may be a word document movedto the left six inches by virtue of a user's gesture interaction withthe word document. A gesture may be any body movement (e.g., head, eyes,hand). An image sensor may capture the gesture and a processor mayinterpret the gesture as an indication of the six inch leftwardmovement. That indication determined by the processor may constitute asecond input.

Some disclosed embodiments may involve analyzing the image data toidentify hand gestures indicative of a desired movement of the virtualobject, and causing the change in the display of the second set ofvirtual objects in a manner corresponding with the desired movement ofthe virtual object. As alluded to in the preceding paragraph, a gesturemay be a movement of a user's hand, or part of a user's hand, that mayindicate or express an intention to cause an action. Such gestures mayinclude, for example, scrolling, pinching, tapping, and/or pressingusing one or more fingers, and or other combinations of movementsinvolving one or more fingers, wrist, and/or a forearm of the user. Animage sensor may create images of a user performing hand gestures.Images may be analyzed to determine if a user may be attempting toperform a specific movement with the virtual object. Images may becompared to collected images to determine a hand gesture of a user.Collected images may be stored and accessed by the processor andcompared using a machine learning module, segmentation, patternmatching, or other techniques. The images may be compared to look forsimilar hand motions. For example, an image of a user's hand gesture mayinclude a user's fingers swiping left. The image may be compared to acollected image of fingers swiping left that corresponds to a desiredmovement of moving an object to the left. Based on the comparison, itmay be determined that the user is attempting to move the object to theleft. Specific hand gestures may relate to specific movements. Forexample, a scrolling gesture may indicate that a user may be trying tomove a virtual object up or down. As another example, a pinching gesturemay indicate that a user may be trying to zoom in on a virtual object.Based on a gesture, a display of a second set of virtual objects maychange. For example, a user may use a pinching gesture to indicate adesire to zoom in on a virtual object. In response, the second set ofvirtual objects may be displayed in a zoomed in or magnified form.

In response to receiving the second input, some disclosed embodimentsmay involve, causing a change in a display of the second set of virtualobjects in a manner corresponding the movement of the item whilemaintaining the first positions of the first set of virtual objects. Achange in a display of the second set of virtual objects may be made ina manner similar to the change in display of the first set of virtualobjects described above. A second set of virtual objects may be locatedon a same virtual plane as an item and a first set of virtual objectsmay be located on a different virtual plane. The item may changelocations on the virtual plane and create an input. A display of asecond set of virtual objects may be changed based on the second inputwhile a display of a first set of virtual objects may not be changed.For example, when an item is moved three feet to the right in the secondvirtual plane, a second set of virtual objects may also move three feetto the right in the second virtual plane, while the first set of virtualobjects may not move in the first virtual plane.

By way of example, FIG. 67 illustrates an example of a movement of anitem and virtual objects relative to the situation illustrated in FIG.65, consistent with some embodiments of the present disclosure. In someembodiments, physical object 6711-and first set of virtual objects 6712may be associated with first virtual plane 6710. For example, first setof virtual objects 6712 may be docked to first virtual plane 6710. Item6721 and second set of virtual objects 6722 and 6723 may be associatedwith second virtual plane 6720. For example, second set of virtualobjects 6722 and 6723 may be docked to second virtual plane 6720. Item6721 may be moved from its original position (shown as 6725) to a newposition (shown as 6726). Second set of virtual objects 6722 and 6723may be moved to correspond with the movement of item 6721. However,first set of virtual objects 6712 may not move.

In some embodiments, causing the change in the display of the first setof virtual objects in response to receiving the first input may includemoving the first set of virtual objects in the manner corresponding themovement of the physical object, and causing the change in the displayof the second set of virtual objects in response to receiving the secondinput includes moving the second set of virtual objects in the mannercorresponding the movement of the item. As described above, a first setof virtual objects may move locations in proportion to a movement of aphysical object. As also described above, a second set of virtualobjects may move locations in proportion to a movement of an item. Forexample, a physical object may be moved three inches to the left of thephysical object's original position. In response, a first set of virtualobjects may also move three inches to the left of the first set ofvirtual object's original position. As another example, an item may movethree inches to the right of the item's original position. In response,a second set of virtual objects may also move three inches to the rightof the second set of virtual object's original position.

In some embodiments, causing the change in the display of the first setof virtual objects in response to receiving the first input may includechanging at least one visual property of the first set of virtualobjects, and causing the change in the display of the second set ofvirtual objects in response to receiving the second input includeschanging at least one visual property of the second set of virtualobjects. A visual property may be a characteristic appearance of anobject as described above, and as provided in examples, above. A changein display may include a change in visual property. For example, aphysical object may move three inches to the left of the physicalobject's original position. In response, a first set of virtual objectsmay reduce in size. As another example, an item may move three inches tothe right of the item's original position. In response, a second set ofvirtual objects may increase in size. Or, a movement of a virtual objectmay result in a re-rendering of the object to reflect a new perspectiveof the object associated with the new location.

In some embodiments, the item may be a virtual object and the movementof the virtual object includes a modification in at least one ofdimensions or an orientation of the virtual object, and wherein theoperations further include changing at least one of dimensions ororientations of the second set of virtual objects in a mannercorresponding the modification of the virtual object. Changes indimension and orientation may include changing physical characteristicsof an item or a virtual object. Dimensions of an item or a virtualobject may be modified by changing the length, breadth, depth, height,or other dimensions of the item. Orientation of an item or a virtualobject may be modified by changing the relative position of the item orvirtual object, respectively. A second set of virtual objects may bemodified in the same way as an item. For example, a length of an itemmay be decreased. In response, respective lengths of a second set ofvirtual items may also be decreased. As another example, an orientationof an item may be modified to be facing a north direction. In response,the second set of virtual items may also be modified to be facing thenorth direction.

In some embodiments, the extended reality environment may include avirtual object associated with the first virtual plane and docked to theitem. A virtual object may be docked to an item as described above. Avirtual object may be located in a first virtual plane of a physicalobject. For example, a first virtual plane may be a virtual planeextending in a horizontal direction. A virtual object may be located ona first virtual plane and also located in a horizontal direction. Avirtual object may be connected to an item located in a second virtualplane.

In response to receiving the first input, some disclosed embodiments mayinvolve causing the change in the display of the first set of virtualobjects in a manner corresponding the movement of the physical objectwhile maintaining a position of a display of the virtual object. Acorresponding change may, for example, include a proportional change interms of distance, or a perspective change in terms of vantage point.When movement of the physical object is detected, a correspondingmovement of the first set of virtual objects may be affected by theprocessor. However, another virtual object may remain unchanged in itsposition. In other words, the positions of only a docked set of virtualobjects may change positions in response to the physical objectmovement, while an undocked virtual object (or virtual objects docked tosomething other than that physical object) remain in place.

In response to receiving the second input, some disclosed embodimentsmay involve causing the change in the display of the second set ofvirtual objects and changing the display of the virtual object in amanner corresponding the movement of the item. Similar to thedescription above, since the second set of virtual objects are docked tothe item (the movement of which constitutes the second input) thedisplay of the second set of virtual objects change when the associateditem moves. The change in the display of the second set of virtualobjects may be proportional to the change in the movement of the item,as described earlier.

In some embodiments, the extended reality environment may include avirtual object associated with the second virtual plane and docked tothe physical object. A virtual object may be docked to a physical objectas described above. A virtual object may be located in a second virtualplane of an item. For example, a second virtual plane may be a virtualplane extending in a vertical direction. A virtual object may be locatedon a second virtual plane and located in a vertical direction. A virtualobject may be connected to a physical object in a first virtual plane.

Some disclosed embodiments may involve, in response to receiving thefirst input, causing the change in the display of the first set ofvirtual objects and a change in a display of the virtual object in amanner corresponding the movement of the physical object. A first set ofvirtual objects and a virtual object may be located in different virtualplanes. A display of a first set of virtual objects may be changed basedon a movement of a physical object in a first virtual plane. A virtualobject may be docked to the physical object and also move in the secondvirtual plane. Movement of the virtual object may be in the same manneras the physical object described above. For example, the physical objectmay move five inches. The first set of virtual objects may also move 5inches. The virtual object may also move five inches due to the virtualobject being docked with the physical object.

Some disclosed embodiments may involve, in response to receiving thesecond input, causing the change in the display of the second set ofvirtual objects in a manner corresponding the movement of the item whilemaintaining a position of the display of the virtual object. Since thevirtual object is not docked to the item, when the second input isreceived, only the display of the docked second set of virtual objectsmay change, while the display of the undocked virtual object may notchange. Similar to the description above, the virtual object may belocated in the second virtual plane and docket to the physical object.The item and the second set of virtual objects may also be located inthe second virtual plane. The item may move and cause a change in thesecond set of virtual objects, but may not cause a change in the virtualobject. For example, an item may move five inches in an upwarddirection. The second set of virtual objects may also move five inchesin an upward direction. The virtual object may not move due to thevirtual object being docked with the physical object.

In some examples, an indication that a physical element is located at aparticular location in the extended reality environment may be received.In some examples, image data captured using an image sensor included inthe wearable extended reality appliance may be received. For example,the image data may be received from the image sensor, from the wearableextended reality appliance, from an intermediate device external to thewearable extended reality appliance, from a memory unit, and so forth.The image data may be analyzed to detect the physical element in theparticular location in the extended reality environment. In anotherexample, radar, Lidar or Sonar sensors may be used to detect thepresence of the physical element at the particular location in theextended reality environment. In some examples, the change in thedisplay of the second set of virtual objects in the manner correspondingthe movement of the item may be selected based on the physical elementbeing located in the particular location. In one example, maintainingthe position of a particular virtual object of the second set of virtualobjects on the second virtual plane may cause the particular virtualobject to collide with the physical element, and in response theparticular virtual object may be moved to a new position (for example,on the second virtual plane, outside the second virtual plane, etc.) sothat the new position of the particular virtual object does not collidewith the physical element, or the docking of the particular virtualelement to the second virtual plane may be canceled or withheld. Inanother example, maintaining the position of the particular virtualobject on the second virtual plane may cause the particular virtualobject to be at least partly hidden by the physical element (which maybe determined using ray casting algorithm), and in response the positionof the particular virtual object on the second virtual plane may bechanged so that at the new position the particular virtual object is nothidden (fully or partially) by the physical element, or the docking ofthe particular virtual element to the second virtual plane may becanceled or withheld.

FIG. 68 illustrates a flow chart of an exemplary method 6800 that may beexecuted by a processor to perform operations for implementing selectivevirtual object display changes. Method 6800 may include a step 6811 ofgenerating an extended reality environment, wherein the environmentincludes a first virtual plane associated with a physical object and asecond virtual plane associated with an item, the second virtual planeextending in a direction transverse to the first virtual plane. Method6800 may also include a step 6812 of accessing a first instruction fordocking a first set of virtual objects in first positions associatedwith the first virtual plane. Further, method 6800 may include a step6813 of accessing a second instruction for docking a second set ofvirtual objects in second positions associated with the second virtualplane. Method 6800 may include a step 6814 of receiving a first inputassociated with a movement of the physical object. Method 6800 mayinclude a step 6815 of causing a change in a display of the first set ofvirtual objects while maintaining the second set of virtual objects insecond positions. Method 6800 may also include a step 6716 of receivinga second input associated with a movement of the item. Further, in someexamples, method 6800 may include an optional step 6817 of causing achange in a display of the second set of virtual objects whilemaintaining the first positions of the first set of virtual objects.

Some embodiments involve determining a display configuration forpresenting virtual content. Determining a display configuration mayinclude determining operating parameters and instructions forconfiguring a display of virtual content. Consistent with theseembodiments and as described below, the display configuration may bedetermined based on retrieved display settings associated with aparticular input device and a value of at least one use parameter. Thedisplay configuration may include, for example, instructions related tothe perspective from which virtual content is presented (e.g., viewingangle, size, position, and/or aspect ratio), instructions related to thepresentation of virtual screens (e.g., a number of virtual screens, asize of a virtual screen, an orientation of a virtual screen, or aconfiguration of a border of a virtual screen), instructions related tothe appearance of the virtual content (e.g., an opacity of the virtualcontent, a color scheme of the virtual content, or a brightness level ofthe virtual content), instructions related to the type of content beingdisplayed (e.g., an operating system for the virtual content, aselection of startup applications, a selection of startup virtualobjects, or a placement of the selected startup virtual objects in anextended reality environment), instructions related to the distance fromthe wearable extended reality appliance for presenting the virtualcontent and/or any other parameter relating to layout or configuration.Additional disclosure and examples of the display configuration aredescribed below in greater detail.

Presenting the virtual content based on a display configuration mayinclude displaying virtual content to a wearer of an extended realityappliance, according to determined display settings. In an example,presenting the virtual content based on a display configuration mayinclude at changing default display settings. For example, changingleast one or more of: the number of virtual screens, the color scheme,the operating system, the brightness level, the opacity level, thestartup virtual objects, or other default display settings. In addition,the presentation may include the display of one or more applications,one or more operating systems, and/or one or more virtual objects whichmay be placed at one or more locations of the virtual content.

In some embodiments, the virtual content presented by the wearableextended reality appliance may include one or more virtual objects, suchas virtual screens. A virtual screen (also referred to as a ‘virtualdisplay’ or a ‘virtual display screen’ herein) may be a virtual objectmimicking and/or extending the functionality of a physical displayscreen, as described above. In one example, each virtual screen maypresent a plurality of user interface elements, such as virtual windows,virtual widgets, or virtual cursors. In some embodiments, each virtualscreen may be configured to display text, visual media, or applications,such as, webpages, videos, pictures, video games, file browsers, emailclients, or web browsers. Consistent with aspects of the presentdisclosure, the presentation of the virtual content may be configured bydisplay settings, for example, the default display settings.

Some embodiments may involve receiving image data from an image sensorassociated with a wearable extended reality appliance. An image sensormay capture image data of the user's environment. In an example, theimage sensor may be built in to the wearable extended reality appliance,for example, in the form of an integrated camera. In another example,the image sensor may be external to the wearable reality appliance, forexample, an external webcam in communication with the wearable extendedreality appliance. The image data may include, for example, an image ofa physical area indicative of the user's visual field includingrepresentations of one or more animate or inanimate objects present inthe visual field. Receiving image data may include a transfer of imagedata from an image sensor to a processing device. In one embodiment, aremote server (e.g., server 210 as depicted in FIG. 2) may receive theimage data from the image sensor, and a wearable extended realityappliance may receive the image data from the remote server.Additionally, or alternatively, an input device (e.g., keyboard 104 asdepicted in FIG. 1, or an input device integrated with a computingdevice) may receive the image data from the image sensor, and thewearable extended reality appliance may receive the image data from theinput device. In some embodiments, the image data may be transmittedover a wireless network, such as Wi-Fi, Bluetooth, near-fieldcommunication, or cellular networks. In some embodiments, the image datamay be transmitted over a wired network, such as LAN, or a USBconnection.

In some embodiments, the wearable extended reality appliance may beconfigured to be paired with multiple input devices. Paring the wearableextended reality appliance with an input device may include establishinga relationship using a pairing mechanism between the wearable extendedreality appliance with the input device. Upon pairing the input deviceto the wearable extended reality appliance, a user may modify thevirtual content displayed by the wearable extended reality applianceusing the input device. A paired connection may be a wired or wirelessconnection established between the wearable extended reality applianceand multiple input devices. In some embodiments, pairing a particularinput device may include using a verification code for each input deviceto establish the connection. In an example, a particular input devicemay be an input device from among the multiple input devices. In anexample, the wearable extended reality appliance may be paired withmultiple input devices over a wireless network, such as Wi-Fi,Bluetooth, near-field communication, or cellular networks, or over awired network, such as LAN, or a USB connection. In an example, thewearable extended reality appliance may be paired with multiple inputdevices over a wired network, such as LAN, or via USB connections.

The multiple input devices may include any type of physical devicesconfigured to receive input from a user or an environment of the user,and to provide the data to a processing device associated with thewearable extended reality appliance. For example, the multiple inputdevices may include at least two or more of: keyboards, mice, styluses,controllers, touchscreens, or other devices that facilitatehuman-computer interaction. In some embodiments, the multiple inputdevices may include at least a first input device and a second inputdevice. The first and second input devices may be of a same type, forexample, the first and second input devices may both be keyboards.Alternatively, the first and second may be of differing types, forexample, the first input device may be a keyboard, and the second inputdevice may be a stylus. In some embodiments, the first and second inputdevices may be visually similar. For example, both the first and secondinput devices may have the similar color, dimensions, patterns, or othersimilar visual identifiers.

In some embodiments, each input device may be associated with defaultdisplay settings. Default display settings may include one or morepreconfigured values that define a configuration or state of a userinterface. In some embodiments, default display settings may be storedon memory associated with each of the multiple input devices. In someembodiments, each input device may include default display settingsunique to the input device's manufacturer or model. In some embodiments,the default display settings may be stored on memory on a remote server.Consistent with the example above, the first and second input devicesmay be associated with different default display settings. For example,the first input device may be associated with first default displaysettings, and the second input device may be associated with seconddefault display settings. In one instance, at least some of the firstdefault display settings may be different from the second defaultdisplay settings.

In some embodiments, the default display settings may include a defaultdistance from the wearable extended reality appliance for presenting thevirtual content. The default distance may be a value, a set of values,or a range within virtual content may be presented to a user. Exampledefault distances for presenting virtual content may include a range ofdistances between 0.5 meters (m) and 7 m. In one example, when thedefault distance is configured to be 1 m, the virtual content mayinitially be displayed at 1 m from the wearable extended realityappliance, and even when the wearable extended reality appliance movesthe distance between the wearable extended reality appliance and thevirtual content remains 1 m. In another example, the virtual content mayinitially be displayed at 1 m from the wearable extended realityappliance, but when the wearable extended reality appliance moves thedistance between the wearable extended reality appliance and the virtualcontent changes accordingly.

In some embodiments, the default display settings may include at leastone of a default number of virtual screens, a default size of a virtualscreen, a default orientation of a virtual screen, or a defaultconfiguration of a border of a virtual screen. The default number ofvirtual screens may be a default or preconfigured number of virtualscreens simultaneously displayed on startup or at a later time. Anexample number of virtual screens may include a number in a rangebetween 1 and 5 virtual screens. A greater number of virtual screens isalso contemplated. The default size of a virtual screen may be a defaultor preconfigured diagonal dimension of a virtual screen. Example virtualscreen sizes may be a range of dimensions between 6″ and 300″. In someembodiments, the default size may be different for each virtual screen.The default orientation may a default or preconfigured orientation ofeach virtual screen. Example default orientations include portrait orlandscape orientations. In an example, each virtual screen may have adifferent default orientation. In some embodiments, the defaultorientation of each virtual screen may be rotated by an offset from aportrait or landscape orientation. For example, a default orientationmay be rotated by 30, 45, or 60 degrees from an initial orientation,such as a landscape orientation. In other embodiment, the defaultorientation of each virtual screen may be indicative of the pitch angle,the yaw angle, and roll angle of each virtual screen. In example, thedefault number of screens may be two, the default size of the firstvirtual horizontal screen is 40″ and the default size of the secondvirtual screen in 20″.

In some embodiments, the default display settings may include a defaultselection of an operating system for the virtual content. The defaultselection of an operating system may be a default or preconfiguredselection of one or more operating systems associated with the wearableextended reality appliance. Examples of operating systems may includeMicrosoft Windows®, Apple MacOS®, Android®, Linux®, UNIX®, and more. Insome embodiments, the default selection may be based on operating systemstability, user's preference, application compatibility, or boot order.In some embodiments, each virtual screen may present a differentoperating system.

In some embodiments, the default display settings may include a defaultselection of startup applications. The default selection of startupapplications may be a selection of one or more startup applications.Example startup applications include operating system processes, suchas, a kernel, window manager, or network drivers. Additionally, oralternatively, example startup applications may include programs, suchas, web browsers, word processors, email clients, chat clients, weatherwidgets, messaging widgets, other virtual widgets, or other executableapplications. In some embodiments, each virtual screen may present adifferent startup application.

In some embodiments, the default display settings may include a defaultselection of startup virtual objects. The default selection of startupvirtual objects may be a selection of one or more virtual objects.Example virtual objects include, virtual cursors, virtual windows,virtual widgets, applications, or other virtual user interface elements.In some embodiments, each virtual screen may present a different virtualobject. In other embodiments, the virtual objects may be presentedoutside the virtual screen. As illustrated in FIG. 1, virtual widgets114A-114D may be displayed next to virtual screen 112 and virtual widget114E may be displayed on table 102. The default selection of startupvirtual objects may include a selection of the type of virtual objectsand their initial placement in the extended reality environment.

In some embodiments, the default display settings may include a defaultplacement of the selected startup virtual objects in an extended realityenvironment. The default placement of the startup virtual objects may bea preconfigured arrangement or location of one or more startup virtualobjects within extended reality environment, e.g., within each virtualscreen. Example placements include a specific coordinate, proximity to avirtual screen boundary, or a center of the virtual screen. In someembodiments, the default placement of the startup virtual objects may berelative to a location of a physical object (e.g., keyboard 104) orrelative to a location of the wearable extended reality appliance. Forexample, virtual widget 114E may be default placed right to keyboard 104and on the same surface.

In some embodiments, the default display settings may include a defaultopacity of the virtual content. The opacity of the virtual content maybe a measure of translucency of the virtual content. The default opacitymay be between a range from 1% to 100%. In one example, 100% opacity mayrepresent an opacity value of the virtual content where less than 10% ofenvironmental light passes through the virtual content. In anotherexample, 100% opacity may represent an opacity value of the virtualcontent where less than 5% of environmental light passes through thevirtual content. In another example, 1% opacity may represent an opacityvalue of the virtual content where greater than 90% of environmentallight passes through the virtual content. In some embodiments, thedefault opacity may be different for each virtual object of the virtualcontent.

In some embodiments, the default display settings may include a defaultcolor scheme of the virtual content. A default color scheme may be apreconfigured coordinated selection of colors for aesthetic orinformational appeal. A default color scheme may be at least one of:monochromatic, achromatic, complementary, split-complementary,analogous, triadic, tetradic, or polychromatic. In some embodiments, thedefault color scheme may be selected to overcome a user's visualimpairment. In some embodiments, each color scheme may include one ormore color palettes. In some embodiments, the default color scheme maybe different for each virtual object of the virtual content.

In some embodiments, the default display settings may include a defaultbrightness level of the virtual content. The default brightness may be apreconfigured luminance value. The default brightness may be a valuebetween a range from 1% to 100%. In some embodiments, the defaultbrightness may be a value between a range from 10 nits to 2000 nits. Inan example, 100% brightness may correspond to about 2000 nits. Inanother example, 100% brightness may correspond to about 200 nits. In anexample 1% brightness may correspond with 10 nits. In another example,1% brightness may correspond to 40 nits. In some embodiments, thedefault brightness may be different for each virtual object of thevirtual content.

In some embodiments, the default display settings may be retrieved fromthe memory. The memory that stores the default display settings may beincluded in any of the component of system 200 depicted in FIG. 2. Forexample, the default display settings may be retrieved from server 210,via a download over communications network 214. In some embodiments, thedefault display settings may be retrieved from the memory of input unit202 via a direct wireless connection, such as an NFC or Bluetoothconnection, between the wearable extended reality appliance and inputunit 202. In yet another example, the default display settings may beretrieved from the memory of input unit 202 via a wired connection, suchas a USB or LAN connection, between the wearable extended realityappliance and input unit 202. In some embodiments, the default displaysettings may be retrieved from memory associated with XR unit 204.

Some embodiments may involve analyzing the image data to detect aparticular input device placed on a surface. The analysis of image datamay include any image processing methods disclosed herein. For example,the analysis of image data may include at least one of: objectdetection, image segmentation, object recognition, or patternrecognition. Consistent with aspects of the present disclosure,detecting the particular input device may involve performing a lookup ina repository of input devices to identify which input device iscurrently being used by the user of the wearable extended realityappliance. The repository of input devices may refer to a data storagecontaining, for example, a table of input devices (e.g., input devicesassociated with the user of the wearable extended reality appliance) andcorresponding default display settings associated with the particularinput device. For example, each input device indicated in the table mayhave a corresponding default display setting stored in the table. In oneexample, the repository may be implemented based on or in a similarmanner as data structure 212. In another embodiment, the analysis mayinclude identify, in the image data, the surface on which the particularinput device is placed. A surface may be a physical surface, such as atable, floor, countertop, coaster, tablemat, mousepad, or other surfacescapable of accommodating an input device. The type of surface on whichthe particular input device is placed may change the default displaysettings. For example, when keyboard 104 is placed on a work desk thewearable extended reality appliance may present virtual contentaccording to a first display configuration, and when keyboard 104 isplaced on a kitchen counter the wearable extended reality appliance maypresent virtual content according to a second display configuration

Some embodiments may involve analyzing the image data to identify anobject in a vicinity of the particular input device. In a manner similarto identifying the particular input device and the surface on which itis placed, the image data may be analyzed to identify other objects. Theother objects may be physical objects such as, other input devices,coffee mugs, computing devices, mobile devices, QR code, barcodes, lightemitters, markers, or display devices. A vicinity may be a distance orradius from the particular input device to the object. Example radii ordistances may be between 1 cm to 10 m. In an example, the object may belocated on the particular input device. In another example, theparticular input device may be placed on the object. In some examples,the image data may be analyzed using at least one of an object detectionalgorithm, an object recognition algorithm or a semantic segmentationalgorithm to identify the object.

Some embodiments may involve determining that the particular inputdevice is the first input device and not the second input device basedon the identification of the object in the vicinity of the particularinput device. In an example, the image data may include a first objectwithin the vicinity of the particular input device, where the firstobject is associated with a first input device. The image data may alsoinclude a second object within a vicinity of another input device, wherethe second object is associated with a second input device. In a mannersimilar to identifying an object in the vicinity of a particular inputdevice, discussed above, the image data may be analyzed to identify oneor more objects. Upon detecting that the first object is within thevicinity of the particular input device, the particular input device maybe identified as a first input device and not a second input device. Forexample, if a QR code is associated with a first input device andlocated on a keyboard, and a coffee mug associated with a second inputdevice is within the vicinity of a tablet, the image analysis maydetermine that the keyboard (e.g., the particular input device) is thefirst input device and not the second input device. In one example, itmay be known that the first input device is placed on a large desk andthat the second input device is placed on a smaller desk. The object maybe the desk that the particular input device is placed on. Measurementsof the object (e.g., the desk that the particular input device is placedon) may be determined by analyzing the image, for example using aregression algorithm or a semantic segmentation algorithm. Thedetermined measurements may be compared with known measurements of thelarge desk and the smaller desk. Based on a result of the comparison,the desk may be identified as the large desk or the small desk, andthereby the particular input device may be identified as the first inputdevice, the second input device, or a completely different input device.

FIG. 69 is an exemplary illustration of a user using a wearable extendedreality appliance. As illustrated in FIG. 69, user 100 may operate XRunit 204, which includes image sensor 472. In this example, the user 100may wear XR unit 204 during operation. The image sensor 472 may bedetect or identify a surface 6910 and an input device 6912 located onthe surface 6910. The image sensor 472 may capture an image includingsurface 6910 and input device 6912 for analysis. A processing device ofsystem 200 may analyze the captured image and identify the particularinput device input device 6912B using repository of input devices 6911.Input device 6912B is associated with a specific set of default displaysettings. Repository of input devices 6911 may be accessible viacommunications network 214, as depicted in the figure or stored in XRunit 204.

Some embodiments may involve determining a value of at least one useparameter for the particular input device. Determining a value of useparameter may include analyzing data from one or more sensors andassigning a value to the use parameter based on the analyzed data. A useparameter may include any characterizing property of the session withwearable extended reality appliance. The use parameters may include oneor more of ergonomic parameters, environmental parameters, or inputdevice parameters. In some embodiments, ergonomic parameters may beindicative of a user's posture or behavior. Example values of ergonomicparameters may include sitting, kneeling, standing, walking, running,dancing, crawling, conversing, gaming, exercising, slouching, bending,stretching, typing, reading, or other activities. Example values ofenvironmental parameters may include the weather, the time of day, theday of week, lighting conditions, temperature, distances to objects inthe environment, noise levels, aromas, a distance between a particularinput device and the wearable extended reality appliance, or othersensory parameters associated with the environment. Example values ofinput device parameters may include battery charge data, deviceidentifiers, pairing status, or other device information.

In some embodiments, the value of at least one use-parameter may bedetermined by analyzing data received from input devices, ambientsensors, positional sensors, accelerometers, gyroscopes, microphones,image sensors, or other sensors. In an example, data fromaccelerometers, gyroscopes, or other positional sensors may be analyzedto determine a value of a user's posture or behavior. In an example,data from ambient sensors, image sensors, positional sensors,microphones, or other environmental sensors may be used to determine avalue of an environmental parameter. In an example, data from inputdevices, image sensors, positional sensors, microphones, or other inputdevice sensors may be used to determine a value of an input deviceparameter.

Some embodiments may involve determining the value of the at least oneuse parameter for the particular input device based on at least one ofan analysis of the image data, data received from the particular inputdevice, or data received from the wearable extended reality appliance.In some embodiments, the analysis of the image data in combination withdata from sensors located on the wearable extended reality appliance,such as a motion sensor, environmental sensor, audio sensor, weightsensor, light sensor, distance sensor, resistive sensor, LIDAR sensor,ultrasonic sensor, proximity sensor, and/or biometric sensor, may beused to compute a distance between the wearable extended realityappliance and the particular input device. In some embodiments, theanalysis of the image data in combination with data from sensors locatedon the particular input device, such as those listed above may be usedto compute a distance between the wearable extended reality applianceand the particular input device.

In an example, proximity sensors may record distance data and an imagesensor may capture image data. The analysis of the distance data andimage data may be used to compute a distance between the wearableextended reality appliance and the particular input device. The distancedata may be as a calibration or standard for distances derived fromanalysis of the image data, thereby allowing improving the accuracy ofdistances computed from the image data, as opposed to the image dataalone.

Some embodiments may involve retrieving from memory default displaysettings associated with the particular input device. In one embodiment,the default display settings associated with the particular input devicemay be retrieved from the memory of input unit 202. For example, thedefault display settings may be retrieved via a direct wirelessconnection, such as an NFC or Bluetooth connection, between the wearableextended reality appliance and input unit 202. In yet another example,the default display settings may be retrieved from the memory of inputunit 202 via a wired connection, such as a USB or LAN connection,between the wearable extended reality appliance and input unit 202. Inanother embodiment, the default display settings associated with theparticular input device may be retrieved from memory associated with XRunit 204. In yet another embodiment, the default display settingsassociated with the particular input device may be retrieved from memoryassociated with server 210.

Some embodiments may involve determining whether the particular inputdevice is a home keyboard or a workplace keyboard. The determination ofwhether the particular input device is a home keyboard or a workplacekeyboard may be based on at least one of: device information,verification codes, such as the visual code or light emitter, visualappearance of the particular input device based on analysis of imagedata, location of the wearable extended reality appliance, time of day,day of the week, or input device utilization. A workplace keyboard maybe an input device that is associated with the user's place ofemployment. A home keyboard may be an input device that is associatedwith the user's domicile. In response to a determination that theparticular input device is a home keyboard, some embodiments may includeretrieving from the memory first default display settings. For example,by reference to FIG. 69, input device 6912B may be determined to be ahome keyboard. In response to a determination that the particular inputdevice is a workplace keyboard, some embodiments may include retrievingfrom the memory second default display settings. For example, byreference to FIG. 69, keyboard 6912C may be determined to be a workplacekeyboard. In some embodiments, the second default display settingsdiffering from the first default display settings. For example, thesecond default display settings may include a selection of startupapplications that are associated with workplace activities, such asproductivity software, development software, VPN, or other workplaceapplications. The first default display settings may include a selectionof startup applications associated with a user's domestic activities,such as video streaming services, music streaming services, smart homeapplications, video games, messaging services (e.g., video chat, SMS,phone calls, instant messaging), or other non-workplace applications. Insome embodiments, the selections need not be mutually exclusive, inother words, some applications may overlap, such as web browsers andVPNs. In some embodiments, each selection of startup applications may beconfigured for a user's workplace and/or domestic activities. In someembodiments, the second default display settings may be adjusted for auser's workplace environment, which may be different from a user'sdomicile. For example, if a user works in a dimly lit undergroundresearch laboratory, but lives in a brightly-lit home, the brightnessand opacity settings in the second default settings may differ from thefirst default display settings to improve visual clarity of thepresented virtual content.

Some embodiments may involve determining whether the particular inputdevice is a private keyboard or a public keyboard. The determination ofwhether the particular input device is a private keyboard or a publickeyboard may be based on at least one of: device information, pairingcodes, such as the visual code or light emitter, visual appearance ofthe particular input device based on analysis of image data, location ofthe wearable extended reality appliance, time of day, day of the week,or input device utilization. A private keyboard may be an input devicethat is owned by the user or associated with the user's personalproperty. A public keyboard may be an input device that is owned andprovided by another party to the user. In some embodiments, a publickeyboard may be a keyboard accessible by other users (e.g., a communalkeyboard). In response to a determination that the particular inputdevice is a private keyboard, some embodiments may include retrievingfrom the memory first default display settings.

In response to a determination that the particular input device is apublic keyboard, some embodiments may include retrieving from the memorysecond default display settings. In some embodiments, the second defaultdisplay settings differing from the first default display settings.

For example, the second default display settings may include a selectionof startup applications that are associated with public activities, suchas productivity software, development software, communications software,VPN, or other collaboration applications. The first default displaysettings may include a selection of startup applications associated witha user's private activities, such as video streaming services, musicstreaming services, smart home applications, video games, or othernon-collaborative applications. In some embodiments, the selections neednot be mutually exclusive, in other words, some applications mayoverlap, such as web browsers or productivity software. In someembodiments, each selection of startup applications may be configuredfor a user's public and/or private activities. In another example, thesecond default display settings may be adjusted for a communal keyboard.For example, if a keyboard for a publicly accessible computer (e.g., ina public library) is detected, an automatic logoff countdown timer maybe among the selection of startup applications of the second defaultdisplay settings. In contrast, if a user's personal laptop keyboard isdetected in a public space (e.g., a coffee shop), the automatic logoffcountdown timer may not be among the selection of startup applicationsof the first default display settings.

Some embodiments may involve determining whether the particular inputdevice is a keys-based keyboard or a touch screen-based keyboard. Thedetermination of whether the particular input device is a keys-basedkeyboard or a touch screen-based keyboard may be based on at least oneof: device information, pairing codes, such as the visual code or lightemitter, visual appearance of the particular input device based onanalysis of image data, location of the wearable extended realityappliance, time of day, day of the week, or input device utilization. Akeys-based keyboard may be an input device that includes and utilizesphysical keys as the primary means of input, for example, a traditionalkeyboard. A touch-based keyboard may be an input device that includesand utilizes virtual keys as the primary means of input, for example, anon-screen keyboard.

In response to a determination that the particular input device is akeys-based keyboard, some embodiments may include retrieving from thememory first default display settings. In response to a determinationthat the particular input device is a touch screen based keyboard, someembodiments may include retrieving from the memory second defaultdisplay settings. In some embodiments, the second default displaysettings differ from the first default display settings. In someembodiments, the second default display settings may be adjusted for akeys-based keyboard, which may be different from a touch screen based.For example, if a keys-based keyboard is detected, the brightness andopacity settings in the first default settings may be decreased toimprove visual clarity of the presented virtual content and thekeys-based keyboard. In contrast, if a touchscreen-based keyboard isdetected, the brightness and opacity settings in the first defaultsettings may be increased to count for the touch screen's brightness andto improve visual clarity of the presented virtual content and thetouchscreen-based keyboard.

Some embodiments involve determining a display configuration forpresenting virtual content based on the value of the at least one useparameter and the retrieved default display settings. Determining thedisplay configuration may include determining an arrangement of thepresentation virtual content according to display settings and the valueof the at least one use parameter, or determining a modification of thedefault display settings associated with the particular input devicebased on the value of the at least one use parameter. Accordingly, thecombination of the retrieved default display settings and the value ofat least one use parameter may be used to determine a displayconfiguration. In an example, a display configuration associated with ause parameters, such as low battery status data from an input device,may include modification of some of the default display settings, forexample, a changed color scheme, a modified selection of virtualobjects, modified startup applications, or other modificationsassociated with a low battery status data on an input device. Thepresentation of the virtual content may be performed in a manner similarto that of presenting the virtual content, as discussed earlier.

Consistent with some embodiments, determining the display configurationmay include modifying the retrieved default display settings based onthe value of the at least one use parameter. For example, if a value ofa use parameter indicates that an input device has a low battery, theretrieved default display settings may be modified to lower thebrightness of the virtual content. In another example, if a value of ause parameter indicates that an input device has a low battery, thedefault display settings may be modified to present a battery monitorassociated with the input device as a startup application. In anotherexample, if a value of a use parameter indicates that the wearableextended virtual reality appliance is operating in bright lightingconditions, the retrieved default display settings may be modified toincrease the opacity of the virtual content. In another example, if avalue of a use parameter indicates a sitting user posture, the defaultdisplay settings may be modified to present a selection a startupapplication associated with the user's workplace. In another example, ifa value of a use parameter indicates a walking user posture, theretrieved default display settings may be modified to decrease theopacity of the virtual content, thereby allowing the user to see wherethey are walking.

Some embodiments may involve determining the display configuration basedon the value of the at least one use parameter, the default displaysettings, and at least one environmental parameter. For example, if avalue of a use parameter indicates a user sitting in bright lightconditions, the retrieved default display settings may be modified toincrease the opacity of the virtual content. In another example, if avalue of a use parameter indicates that an input device has a lowbattery and the environmental parameter indicates that the input deviceis not near a charging station, the display configuration may bemodified to present a battery monitor associated with the input deviceas a startup application.

In some embodiments, when the at least one use parameter is reflectiveof a distance of the particular input device from the wearable extendedreality appliance, as discussed earlier, some embodiments may includedetermining a first display configuration when the distance is greaterthan a threshold. As discussed earlier, analysis of image data andsensor data of the wearable extended reality appliance may be used todetermine a distance between the wearable extended reality appliance andthe particular input device. When the distance is determined to be morethan a threshold distance, for example one meter, away from a particularinput device, the virtual content may be presented using a first displayconfiguration. In an example, if the user walks more than 1 meter awayfrom their workplace keyboard, the presented virtual content may bere-arranged such that work related virtual screens are not presented.Some embodiments may include determining a second display configurationwhen the distance is less than the threshold. When the distance isdetermined to be less than a threshold distance, for example one meter,away from a particular input device, the virtual content may bepresented using the second display configuration. In an example, if theuser walks to within one meter of their workplace keyboard, the virtualcontent may be re-arranged such that work related virtual screens arepresented. In some embodiments, the second display configuration differsfrom the first display configuration as discussed above.

In some embodiments, when the at least one use parameter is reflectiveof a posture of a user of the wearable extended reality appliance, asdiscussed earlier, some embodiments may include determining a firstdisplay configuration when a first posture is identified. As discussedearlier, sensor data of the wearable extended reality appliance may beused to determine one or more postures of a user wearing the wearableextended reality appliance. When a first posture is identified, forexample sitting, the virtual content may be presented using a firstdisplay configuration. In an example, if the user is identified assitting at a workplace, virtual content may be arranged such that workrelated virtual screens and applications are presented.

Some embodiments include determining a second display configuration whena second posture is identified. When a second posture is identified, thevirtual content may be presented according to a second displayconfiguration. In an example, if the user is identified as conversing(e.g., in a conversation), virtual content may be re-arranged such thatwork related virtual screens and applications are not presented. In someembodiments, the second display configuration differs from the firstdisplay configuration, as discussed earlier.

In some embodiments, when the at least one use parameter is reflectiveof a type of the surface on which the particular input device is placed,as discussed earlier, some embodiments may include determining a firstdisplay configuration when a first type of the surface is identified.When a first type of surface is identified, the virtual content may bepresented using the determined first display configuration. In anexample, a user places their keyboard on a bed, virtual content may bearranged such that entertainment related virtual screens andapplications are presented.

Some embodiments may involve determining a second display configurationwhen a second type of the surface is identified. When a second type ofsurface is identified, the virtual content may be presented using thedetermined second display configuration. In an example, if a user placestheir keyboard on a desk, virtual content may be arranged such thatproductivity related virtual screens and applications are presented. Insome embodiments, the second display configuration differs from thefirst display configuration, as discussed earlier.

In some embodiments, when the at least one use parameter is reflectiveof battery-charge data associated with the particular input device, asdiscussed earlier, some embodiments, may include determining a firstdisplay configuration when the particular input device operates onbatteries. When battery charge data indicates that the particular inputdevice is in a discharging state (e.g., running on battery power or notconnected to an external power source), the virtual content may bepresented using the determined first display configuration. In anexample, if the wearable extended reality appliance is paired with atablet running on battery power, virtual content may be arranged suchthat battery monitoring information related the tablet are included inthe presentation of the virtual content.

Some embodiments may involve determining a second display configurationwhen the particular input device is connected to an external powersource. When battery charge data indicates that the particular inputdevice is in a charging or charged state (e.g., not running on batterypower or connected to an external power source), the virtual content maybe presented using the determined second display configuration. In anexample, if the wearable extended reality appliance is paired with atablet connected to a wall outlet, the virtual content may be arrangedsuch that battery monitoring information related the tablet are notincluded in the presentation of the virtual content. In someembodiments, the second display configuration may differ from the firstdisplay configuration.

Some embodiments include causing a presentation of the virtual contentvia the wearable extended reality appliance according to the determineddisplay configuration.

The presentation of the virtual content may be caused by thedetermination of a display configuration. The presentation of thevirtual content according to a determined display configuration may beexecuted in a manner similar to presenting the virtual content, asdiscussed earlier. The determination of a display configuration maychange the presentation of the virtual content in a variety of ways.

For example, the display configuration may arrange the virtual contentsuch that no content is displayed in the center of the screen while auser wearing the wearable extended reality appliance is in aconversation. Additionally, environmental sensor may detect the lightingconditions and modify the display configuration to adjust the defaultvirtual content's brightness and/or opacity included in the displaysettings associated with the particular input device for improved visualfidelity of the virtual content.

In another example, the display configuration may modify the virtualcontent such that informational map content (e.g., points of interest,business information, customer reviews, commute information) isdisplayed in the center of the screen while a user is walking whilewearing the wearable extended reality appliance. The indication that theuser is walking is use parameter and the inclusion of the informationalmap is change in the default display settings associated with theparticular input device. Additionally, environmental sensor may detectthe lighting conditions and modify the display configuration to adjustthe virtual content's brightness and/or opacity for improved visualfidelity of the virtual content.

FIG. 70 is an exemplary illustration of a display configurationpresented to a user of a wearable extended reality appliance. Asillustrated in FIG. 70, XR unit 204 may determine a displayconfiguration 7000 and present virtual content 7010 based on the valueof the at least one use parameter (e.g., associated with the amount oflight emitted from lamp 7020) and the retrieved default display settings(e.g., associated with particular input device input device 6912B). Inthis example, a user sitting at their home desk with his or her homekeyboard may be presented a display configuration where the virtualcontent 7010 includes an arrangement of widgets 7012A-7012J around anapplication 7014. The widgets 7012A-7012J include a settings widget7012A, a fitness application 7012B, a mail client 7012C, a systemmonitor 7012D, a weather application 7012E, a photo viewer 7012F, astock tracker 7012G, a time utility 7012H, a messaging client 70121, anda news application 7012J.

FIG. 71 is an exemplary illustration of a display configurationpresented to a user of a wearable extended reality appliance. Asillustrated in FIG. 71, XR unit 204 may determine a displayconfiguration 7100 and present virtual content 7110 based on the valueof the at least one use parameter (e.g., associated with the amount oflight emitted from lamp 7120) and the retrieved default display settings(e.g., associated with particular input device input device 6912A). Inthis example, a user sitting at their workplace desk with his or herworkplace keyboard may be presented a display configuration where thevirtual content 7110 includes an arrangement of widgets 7112A-7112Daround an application 7114. The widgets 7112A-7112D include a settingswidget 7112A, a mail client 7112B, a messaging client 7112C, and aweather utility 7112D.

Some embodiments may involve pairing the particular input device to thewearable extended reality appliance. The particular input device mayinclude at least one of: keyboards, mice, styluses, controllers, touchscreens, or other devices that facilitate human-computer interaction.The pairing may be performed in a manner similar to paring multipledevices, as discussed earlier.

Some embodiments may involve accessing stored information associatingthe multiple input devices with different default display settings. Thestored information associating multiple input devices with differentdefault display settings may be accessed from a remote server, via adownload over a network. In some embodiments, the stored information maybe accessed from the memory of the input device via a direct wirelessconnection, such as an NFC or Bluetooth connection, between the wearableextended reality appliance and the input device. In yet another example,the stored information may be accessed from the memory of the inputdevice via a wired connection, such as a USB or LAN connection, betweenthe wearable extended reality appliance and the input device. In someembodiments, the stored information may be accessed from memoryassociated with the wearable extended reality appliance.

Some embodiments may involve retrieving from the accessed storedinformation and the default display settings associated with the pairedparticular input device. The default display settings associated withthe paired particular input device may be retrieved from the accessedstored information on a remote server, via a download over a network. Insome embodiments, the default display settings associated with thepaired particular input device may be retrieved from the storedinformation on memory of the input device via a direct wirelessconnection, such as an NFC or Bluetooth connection, between the wearableextended reality appliance and the input device. In yet another example,the default display settings associated with the paired particular inputdevice may be retrieved from the stored information on memory of theinput device via a wired connection, such as a USB or LAN connection,between the wearable extended reality appliance and the input device. Insome embodiments, the default display settings associated with thepaired particular input device may be retrieved from the storedinformation on memory associated with the wearable extended realityappliance.

In some embodiments, the pairing of the particular input device to thewearable extended reality appliance is based on a detection of a visualcode depicted in the image data. In some embodiments, the verificationcode may be a visual code. The visual code may include at least one of:a barcode, QR code, an alphanumeric access code, or any other uniquevisual indicator. In an example, the visual code may be detected oridentified using image analysis of the image data. Upon detection, thevisual code may cause the wearable extended reality appliance to executeone or more instructions to enable pairing of the particular inputdevice to the wearable extended reality appliance.

In some embodiments, the visual code may be in the vicinity of theparticular input device. For example, the visual code may be within apredetermined distance of the particular input device. The predetermineddistance may be a range of distances between 1 mm and 2 m. In someembodiments, the visual code may be located on the particular inputdevice. The pairing of the particular input device to the wearableextended reality appliance may be similar to that of pairing multipleinput devices to the wearable extended reality appliance discussedearlier. In an example, the wearable extended reality appliance may bepaired with a particular input device over a wireless network, such asWi-Fi, Bluetooth, near-field communication, or cellular networks. In anexample, the wearable extended reality appliance may be paired with aparticular input device over a wired network, such as LAN, or a USBconnection.

In some embodiments, the pairing of the particular input device to thewearable extended reality appliance is based on a detection of lightemitted by a light emitter included in the particular input device andcaptured by a sensor included in the wearable extended realityappliance. A light emitter may include at least one of: a LED, IRemitter, UV light emitter, a monochromatic light emitter, anincandescent bulb, a fluorescent bulb, neon light tubes, or otherartificial light emitters. Detection of the light emitter may includeimage analysis of image data that includes the light emitter. Upondetection, the wearable extended reality appliance may execute one ormore instructions to enable pairing of the particular input device tothe wearable extended reality appliance. In some embodiments, the lightemitter may be in the vicinity of the particular input device. Forexample, the light emitter may be within a predetermined distance of theparticular input device. The predetermined distance may be a range ofdistances between 1 mm and 2 m. In some embodiments, the light emittermay be located on the particular input device. The pairing of theparticular input device to the wearable extended reality appliance maybe similar to that of pairing of the particular input device to thewearable extended reality appliance is based on a detection of a visualcode depicted in the image data discussed earlier.

FIG. 72 provides a flowchart of an example method 7200 for determining adisplay configuration for presenting virtual content executed by aprocessing device of system 200 as illustrated in FIG. 2. The processingdevice of system 200 may include a processor within a mobilecommunications device (e.g., a mobile communications device 206), aprocessor within a server (e.g., server 210), a processor within awearable extended reality appliance, or a processor within an inputdevice associated with a wearable extended reality appliance (e.g.,keyboard 104). It will be readily appreciated, that variousimplementations are possible and that any combination of components ordevices may be utilized to implement the example method. It will also bereadily appreciated that the illustrated method can be altered to modifythe order of steps, delete steps, or further include additional steps,such as steps directed to optional embodiments. In step 7212, method7200 may include receiving image data from an image sensor associatedwith a wearable extended reality appliance. In step 7214, method 26-2-00may include analyzing the image data to detect a particular input deviceplaced on a surface. In step 7216, method 26-2-00 may includedetermining a value of at least one use parameter for the particularinput device. In step 7218 method 7200 may include retrieving frommemory default display settings associated with the particular inputdevice. In step 7220, method 7200 may include determining a displayconfiguration for presenting virtual content based on the value of theat least one use parameter and the retrieved default display settings.In step 7222, method 7200 may include causing a presentation of thevirtual content via the wearable extended reality appliance according tothe determined display configuration.

Some disclosed embodiments may involve augmenting a physical displaywith an extended reality display. A physical display may include anydevice capable of converting electrical signals into visual imagery. Forexample, a physical display may include, a screen in an informationterminal display, a desktop computer display, a laptop computer display,a mobile phone display, a smartphone display, a tablet personal computerdisplay, an information kiosk display, an ATM display, a vehicledisplay, a medical device display, a display of a system for financialtransactions, a display of a mobile game console, a projector, atelevision, a display of an ultra-mobile personal computer, a wearabledisplay, and any other physical surface on which visual information ispresented.

In some embodiments, the physical display may include, for example, amonitor using a liquid crystal display, plasma technology, cathode raytube, light emitting diodes, holographic display, or any other type ofoutput device that displays information in pictorial or text form. Byway of example, as illustrated in FIG. 73, physical display 7310 may bein the form of a computer monitor. In another example, as illustrated inFIG. 74, physical display 7410 may be included in a smart watch.

In some embodiments, the physical display may be a part of a handheldcommunications device. A handheld communications device, such as amobile communications device 206 illustrated in FIG. 2, may be acomputer small enough to hold and operate in the hand. Typically, ahandheld communications device may include an LCD or OLED flat screeninterface, or any other display providing a touchscreen interface. Thehandheld communications device may also include digital buttons, adigital keyboard; and/or physical buttons along with a physicalkeyboard. Such devices may connect to the Internet and interconnect withother devices such as car entertainment systems or headsets via Wi-Fi,Bluetooth, cellular networks or near field communication (NFC). Handheldcommunications devices may include (i) Mobile computers such as a tabletcomputer, a netbook, a digital media player, an enterprise digitalassistant, a graphing calculator, a handheld game console, a handheldPC, a laptop, a mobile internet device (MID), a personal digitalassistant (PDA), a pocket calculator, a portable media player, and anultra-mobile PC; (ii) mobile phones such as camera phones, featurephones, smartphones, and phablets, (iii) digital cameras such as adigital camcorder, a digital still camera (DSC), a digital video camera(DVC), and a front-facing camera, (iv) pagers, (v) personal navigationdevice (PND), (vi) wearable computers such as a calculator watch, asmartwatch, and a head-mounted display, and (vii) smart cards.

In some embodiments, the physical display may be a part of a stationaryappliance. A stationary appliance may be a device that is not typicallymoved during operation. Stationary appliances may include desktoppersonal computers, televisions, refrigerators, ovens, washers, dryers,dishwashers, kiosks, automated teller machines (ATM), checkoutregisters, voting machines, gambling machines, and gasoline pumps.

In some embodiments, the physical display may be a part of an inputdevice configured to generate text to be presented on the physicaldisplay. An input device may include a physical device configured toreceive input from a user or an environment of the user, and to providethe data to a computational device, as described above. In an inputdevice configured to generate text, the data may include textual data.Some non-limiting examples of input devices configured to generate textmay include a physical keyboard, a virtual keyboard, a touchscreenconfigured to provide a virtual keyboard to a user, a microphoneintegrated with a computing device configured to use a speechrecognition algorithm to generate text from audio captured using themicrophone, and so forth. In some examples, the input device may beintegrated with other electrical components, such as a computing deviceor a physical display, for example in a single housing. In someembodiments, the physical display may be a part of an input deviceconfigured to generate text to be virtually presented in the virtualspace.

By way of example, as illustrated in FIG. 73, virtual space 7320 mayprovide a visual display region that may at least partially surround thephysical display 7310. In another example, as illustrated in FIG. 74,virtual space 7420 may provide a visual display region that may at leastpartially surround the physical display 7410.

Augmenting with an extended reality display, for example, may includeproviding additional visual real estate on a virtual space that isbeyond that of a physical display. In some embodiments, this mayinclude, for example, virtually presenting text, images, or otherinformation in an extended reality space (e.g., display generated by orviewable using an extended reality appliance) in addition to text,images, or other information provided on a physical display. In someembodiments, this may include, for example, moving text, images, orother information from a physical display to a virtual presentation in avirtual space outside the physical display.

Some disclosed embodiments may involve performing operations includingreceiving first signals representative of a first object fully presentedon a physical display. In some embodiments, the first signals may bedigital signals or analog signals. Digital signals may refer to a seriesof transmissible digital that convey information. In one example, thefirst signals may represent, for example, sensor data, textual data,voice data, video data, graphical data, geometric data, or any otherform of data that provides perceptible information associated with thefirst object. In another example, the first signals may include anindication that the first object is fully presented on the physicaldisplay. In some examples, receiving the first signals may include atleast one of reading the first signals from memory, receiving the firstsignals from an external device, receiving the first signals from asoftware or hardware component controlling at least part of thepresentation on the physical display, or receiving the first signalsfrom a software or hardware component controlling an extended realityenvironment including the virtual space.

In some embodiments, first signals may be received from an operatingsystem controlling a physical display. First signals may be received,for example, via any electromagnetic communication that is transmittedwirelessly or via a wire, via a memory unit, via a communication bus,and so forth. An operating system may include, for example, systemsoftware that manages computer hardware, software resources, or providesany other common services for computer programs. The operating systemmay control, directly or indirectly, on different aspects of thephysical display. For example, the operating system may control at leastone of a frame rate of the physical display, a display resolution of thephysical display, a color scheme of the physical display, a displaybrightness of the physical display, a display contrast of the physicaldisplay, or any other control parameter of the physical display. Inanother example, the operating system may control the content displayedby the physical display, for example by rendering the content, byproviding the content (for example, through a video card, through ashared memory, through a communication cable, etc.), and so forth.

In some embodiments, first signals may be received from a pointingdevice associated with the wearable extended reality appliance. Apointing device may include all possible types of devices and mechanismsfor inputting two-dimensional or three-dimensional information. Examplesof pointing input devices may include a computer mouse, trackball,touchpad, trackpad, joystick, pointing stick, stylus, light pen, or anyother physical or virtual input mechanism. For example, the clicking anddragging of a computer mouse may generate the first signal that couldmove an icon visible in a VR headset.

Some disclosed embodiments may involve receiving first signalsrepresentative of a first object fully presented on a physical display.In one example, the first object may be an incidence of virtual content.In some examples, the first object may be or include an element of auser interface, such as a window, an input control element, a navigationcomponent, an information component, an icon, a widget, and so forth. Insome examples, the first object may be or include at least one of text,image, video or graphical element (such as a two-dimensional graphicalelement, a two-dimensional projection of a three-dimensional graphicalelement, and so forth).

An object may be fully presented on a physical display by for example,providing the totality of the visible representation of the objectwithin the visual boundaries of, for example, the physical display. Forexample, a representation of a square on a computer monitor that isfully presented, may result in the computer monitor displaying all foursides and all four corners of the square on the computer monitor.

In some embodiments, the first object may include at least one of awidget or an icon of an application. A software widget may be atask-oriented application or component. A desk accessory or applet maybe example of a simple, stand-alone widget, in contrast with a morecomplex application such as a spreadsheet or word processor. Thesewidgets are typical examples of transient and auxiliary applicationsthat don not necessarily monopolize the user's attention. Additionally,graphical control elements (GUI “widgets”) are examples of reusablemodular components that are used together to build a more complexapplication, allowing programmers to build user interfaces by combiningsimple, smaller components. An icon may be a pictogram or ideogramdisplayed on a display in order to help the user navigate a computersystem. The icon itself may be a quickly comprehensible symbol of asoftware tool, function, or a data file, accessible on the system andmay be more like a traffic sign than a detailed illustration of theactual entity it represents. An icon may serve as an electronichyperlink or file shortcut to access the program or data.

By way of example, as illustrated in FIG. 73, the first objects 7330,7340 may be in the form of widgets 7330 or an icon 7340, respectively.As seen in FIG. 73, the first objects 7330, 7340, for example, virtualobjects in the form of widgets or icons may be completely visible withinthe boundaries of physical display 7310. In another example, asillustrated in FIG. 74, the first objects 7430, 7440 may be in the formof a widget or an icon 7440. As seen in FIG. 74, the first object 7430,7440, for example, in the form of the widget 7430 or the icon 7440 maybe completely visible within the boundaries of physical display 7410.

Some disclosed embodiments may involve performing operations includingreceiving second signals representative of a second object having afirst portion presented on the physical display and a second portionextending beyond a border of the physical display. The second signals,like the first signals, may be digital signals or analog signals. In oneexample, the second signals may represent, for example, sensor data,textual data, voice data, video data, graphical data, geometric data, orany other form of data that provides perceptible information associatedwith the second object. In another example, the second signals mayinclude an indication that the second object has a first portionpresented on the physical display and a second portion extending beyonda border of the physical display. In yet another example, the secondsignals may include information associated with the first portion and/orthe second portion of the second object. In some examples, receiving thesecond signals may include at least one of reading the second signalsfrom memory, receiving the second signals from an external device,receiving the second signals from a software or hardware componentcontrolling at least part of the presentation on the physical display,or receiving the second signals from a software or hardware componentcontrolling an extended reality environment including the virtual space.In some embodiments, the second object, like the first object, may be anincidence of virtual content. In some examples, the second object may beor include an element of a user interface, such as a window, an inputcontrol element, a navigation component, an information component, anicon, a widget, and so forth. In some examples, the second object may beor include at least one of text, image, video or graphical element (suchas a two-dimensional graphical element, a two-dimensional projection ofa three-dimensional graphical element, and so forth). In someembodiments, the second signals may be received from an operating systemcontrolling the physical display, for example in a manner similar tothat described above in relation to the first signals. In otherembodiments, the second signals may be received from a pointing deviceassociated with the wearable extended reality appliance, for example ina manner similar to that described above in relation to the firstsignals.

A first portion of an object may include, for example, a fractional partof the totality of a graphical or a visual representation of the object.This fractional part may correspond to any percentage of the secondobject less than one hundred percent and greater than zero percent. Forexample, a representation of cross may include a lower fractional partof the whole that has a T-shape within the visual boundaries of, forexample, the physical display. A second portion of an object mayinclude, for example, a remaining fractional part of the totality of thegraphical or the visual representation of the object (for example, whenthe first portion or at least part of the first portion is subtractedout). This fractional part may correspond to any percentage of thesecond object less than one hundred percent and greater than zeropercent. For example, a representation of cross may include a lowerfractional part of the whole that has a T-shape and a remaining upperfractional part that has an inverted T-shaped. In this example, thelower T-shaped fractional part and the upper inverted T-shapedfractional part may represent the entirety of the cross-shaped whole.

By way of example, as illustrated in FIG. 73, the second object 7350 maybe in the form of a widget. As seen in FIG. 73, the second object 7350may include a first portion 7352 that is viewable within physicaldisplay 7310 and a second portion 7354 that extends beyond the boundaryof physical display 7310. In another example, as illustrated in FIG. 74,the second object 7450 may be in the form of an icon 7450. As seen inFIG. 74, the second object 7450, for example, in the form of the icon7450 may include a first portion 7452 that is viewable within physicaldisplay 7410 and a second portion 7454 that extends beyond the boundaryof physical display 7410.

In some embodiments, only the first portion of an object may bepresented on the physical display. In the above-described example wherethe second object may be a representation of cross, only part of thecross may be visible on the physical display. In this example, thecross-shaped second object may be positioned toward the top of thephysical display such that only the first portion (e.g., the T-shapedfractional part) is visible on the physical display. As a result of thisarrangement, the second portion may extend beyond a border of thephysical display. In the above-described example where the second objectmay be a representation of cross, the remaining part of the cross maynot be visible on the physical display. In this example, thecross-shaped second object may be positioned toward the top of thephysical display such that only the second portion (e.g., the invertedT-shaped fractional part) is not visible on the physical display. Insome examples, when only the first portion of the object is presented onthe physical display, the second portion of the object may be invisible,may not be presented at all, may be presented in a virtual space via oneor more wearable extended reality appliances (and therefore may bevisible to users of the one or more wearable extended reality appliancesand may be invisible to other people), may be otherwise displayed, andso forth.

In some embodiments, the second object may include at least one of awidget or an icon of an application, for example as described above inrelation to the first object. By way of example, as illustrated in FIG.73, the second object 7350 may be in the form of a widget. As seen inFIG. 73, the second object 7350 may include a first portion 7352 that isviewable within physical display 7310 and a second portion 7354 thatextends beyond the boundary of physical display 7310. In anotherexample, as illustrated in FIG. 74, the second object 7450 may be in theform of an icon 7450. As seen in FIG. 74, the second object 7450 mayinclude a first portion 7452 that is viewable within physical display7410 and a second portion 7454 that extends beyond the boundary ofphysical display 7410.

As describes above, second portion 7354 and/or second portion 7454 maynot be displayed at all, may be presented in a virtual space via one ormore wearable extended reality appliances (and therefore may be visibleto users of the one or more wearable extended reality appliances and maybe invisible to other people), or may be otherwise displayed.

In some embodiments, the second object may partially overlap with thefirst object. The term “partially overlap” as used in this disclosuremay include, at least some portion of each object virtually occupyingthe same location. This joint occupation of a location may result inobscuring of one or both portions of these objects. By way of example,as illustrated in FIG. 73, the second object 7350 may partially overlapfirst object 7330 on physical display 7310.

Some disclosed embodiments may involve performing operations includingreceiving third signals representative of a third object initiallypresented on the physical display and subsequently wholly moved beyondthe border of the physical display. The third signals, like the firstand second signals, may be digital signals or analog signals. In oneexample, the third signals may represent, for example, sensor data,textual data, voice data, video data, graphical data, geometric data, orany other form of data that provides perceptible information associatedwith the third object. In another example, the third signals may includean indication that the third object is initially presented on thephysical display and subsequently wholly moved beyond the border of thephysical display. In some examples, receiving the third signals mayinclude at least one of reading the third signals from memory, receivingthe third signals from an external device, receiving the third signalsfrom a software or hardware component controlling at least part of thepresentation on the physical display, or receiving the third signalsfrom a software or hardware component controlling an extended realityenvironment including the virtual space. In some embodiments, the thirdobject, like the first and second objects, may be an incidence ofvirtual content. In some examples, the third object may be or include anelement of a user interface, such as a window, an input control element,a navigation component, an information component, an icon, a widget, andso forth. In some examples, the third object may be or include at leastone of text, image, video or graphical element (such as atwo-dimensional graphical element, a two-dimensional projection of athree-dimensional graphical element, and so forth). In some embodiments,the third signals may be received from an operating system controllingthe physical display, for example in a manner similar to that describedabove in relation to the first signals. In some embodiments, the thirdsignals may be received from a pointing device associated with thewearable extended reality appliance, for example in a manner similar tothat described above in relation to the first signals.

In some embodiments, the third object may be initially presented on thephysical display. This initial presentation may be indicative of apartial presentation or a total presentation of the third object. Forexample, if the third object is an internet browser window, the internetbrowser window may either initially be (i) completely visible on thephysical display or (ii) partially visible on the physical display dueto parts of the browser window that may extend beyond the borders of thephysical display. In one example, the third object may be initiallyfully presented on a physical display and subsequently wholly movedbeyond the border of the physical display. In another example, the thirdobject may initially have one portion presented on the physical displayand another portion extending beyond a border of the physical display,and may subsequently wholly moved beyond the border of the physicaldisplay.

In some embodiments, the third object may be subsequently wholly movedbeyond the border of the physical display. Moving the third objectwholly beyond the border of the physical display may be triggered by auser input (such as a hand gesture pushing or pulling the third object,a cursor dragging the third object, a voice command, and so forth), areception of a communication signal, a detection of an event in theenvironment, or any other type of trigger. This subsequent movement mayinclude translation from a partial presentation or a total presentationof the third object on the physical display to the absence of the thirdobject on the physical display. In the above-described example where thethird object may be an internet browser window, the internet browserwindow may move from being either completely visible on the physicaldisplay or partially visible on the physical display to no part of theinternet browser window being visible on the physical display due to theinternet browser traveling completely beyond the boundaries of thephysical display.

In some embodiments, the third object may include at least one of awidget or an icon of an application, for example as described above inrelation to the first object. By way of example, as illustrated in FIG.73, the third object 7360 may be in the form of a widget. As seen inFIG. 73, the third object 7360 may have initially been in the positionof the first object 7330, 7340 or the second object 7350 has relocatedto a position beyond the borders of physical display 7310. In anotherexample, as illustrated in FIG. 74, the third object 7460, for example,in the form of the icon 7460 that may have initially been in theposition of the first object 7430, 7440 or the second object 7450 hasrelocated to a position beyond the borders of physical display 7410.

In some embodiments, the second object may partially overlap with thethird object. By way of example, as illustrated in FIG. 73, the secondobject 7350 may partially overlap icon 7460 on physical display 7410.

In some embodiments, the first object, the second object, and the thirdobject may be concurrently presented on the physical display and in thevirtual space. By way of example, as illustrated in FIG. 73, firstobject 7330, 7340, the second object 7350 may partially overlap icon7460 on physical display 7410.

In response to receipt of the second signals, some disclosed embodimentsmay involve causing the second portion of the second object to bepresented via a wearable extended reality appliance in a virtual spacewhile the first portion of the second object is presented on thephysical display. In some embodiments, the second portion of the secondobject may be caused via a wearable extended reality appliance to bepresented in a virtual space. This presentation may be indicative of apartial presentation of the second object. For example, if the secondobject is an internet browser window, a right half of the window may bevisible on or within the extended reality appliance. In someembodiments, this presentation of the second portion of the secondobject may occur while the first portion of the second object ispresented on the physical display. This presentation may be indicativeof a partial presentation of the second object. In the above-describedexample where the second object may be an internet browser window, aleft half of the window may be visible on the physical display. In thisexample, the boundary between the left and right portion of the internetbrowser window may correspond to a left border of the physical display.

By way of example, as illustrated in FIG. 73, the second object 7350 maybe in the form of a widget. As seen in FIG. 73, the second object 7350may include a first portion 7352 that is viewable within physicaldisplay 7310 and a second portion 7354 that is viewable within thevirtual space 7320. In another example, as illustrated in FIG. 74, thesecond object 7450 may be in the form of an icon 7450. As seen in FIG.74, the second object 7450, for example, in the form of the icon 7350may include a first portion 7452 that is viewable within physicaldisplay 7410 and a second portion 7454 that is viewable within thevirtual space 7420.

In response to receipt of the third signals, some disclosed embodimentsmay involve causing the third object to be wholly presented via thewearable extended reality appliance in the virtual space following thethird object having been wholly presented on the physical display. Insome embodiments, the third object may be caused via a wearable extendedreality appliance to be wholly presented in a virtual space. Thispresentation may be indicative of a total presentation of the thirdobject. For example, if the third object is an internet browser window,a totality of the window may be visible on within the extended realityappliance. In some embodiments, this presentation of the third objectmay occur following the third object having been wholly presented on thephysical display. This presentation may be indicative of a totalpresentation of the third object. In the above-described example wherethe third object may be an internet browser window, a totality of thewindow may have been initially visible on within the physical display.In this example, the third object may have completely traversed a borderof the physical display to become exclusively visible via the extendedreality appliance.

By way of example, as illustrated in FIG. 73, the third object 7360 maybe in the form of a widget located completely within the virtual space7320. Prior to arriving at this location, the third object 7360 may havebeen initially located in the position of the first object 7330 that iscompletely within physical display 7310.

Some disclosed embodiments may involve performing further operationsincluding receiving image sensor signals representing an image of thephysical display. Image sensor signals may include digital or analogsignals originating from or produced by an image sensor. In someexamples, receiving the image sensor signals may include at least one ofreading the image sensor signals from memory, receiving the image sensorsignals from an external device, or capturing the image sensor signalsusing the image sensor.

Some disclosed embodiments may involve performing further operationsincluding ascertaining a bounding edge of the physical display. Abounding edge of the physical display may refer to refer to a perimeterthat bounds a portion or a totality of the physical display. Such anedge may be ascertained by calculating or determining based on imagesensor data, stored coordinates, or any other information that could beused to establish the bounding edge of the physical display. In someexamples, the received image sensor signals or the image may be analyzedusing a template matching to ascertain a bounding edge of the physicaldisplay. In another example, the received image sensor signals or theimage may be analyzed using a semantic segmentation algorithm todetermine the pixels in the image corresponding to the physical display,thereby ascertaining a bounding edge of the physical display.

Some disclosed embodiments may involve registering the virtual spacewith the physical display based on the ascertained bounding edge.Registering the virtual space with the physical display may refer todefining the bounds of the virtual space, creating subsections of thevirtual space, or any other manipulation of the virtual space that maydelineate the location of the physical display. In the above examplewhere a perimeter defining the borders of the physical display wasextrapolated, that perimeter may be applied to the extended realityspace to, for example, exclude the footprint of the physical displayfrom the area within the wearable extended reality appliance that willgenerate images.

By way of example, as illustrated in FIG. 73, image sensor data may beused to delineate the boundaries of virtual space 7320 to exclude anarea defined by the boundaries of physical display 7310. Accordingly,the third objects 7360 and second portion 7354 of the second object 7350fall within the boundaries of the virtual space 7320 while the firstobjects 7340 and the first portion 7352 of the second object 7350 donot. Additionally, as illustrated in FIG. 74, image sensor data can beused to delineate the boundaries of virtual space 7420 to exclude anarea defined by the boundaries of physical display 7410. Accordingly,the third objects 7460 and second portion 7454 of the second object 7450fall within the boundaries of the virtual space 7420 while the firstobjects 7440 and the first portion 7452 of the second object 7450 donot.

In some embodiments, the physical display may include a frame definingthe bounding edge. Some physical displays (such as computer monitors,smartphones, smartwatches, etc.) may contain a “frame” surrounding theimage producing screen, while other physical displays such as somemodern smartphones may have a “frameless” screen such that the screenextends completely to the edge of the device. By way of example, asillustrated in FIG. 74, physical display 7410 includes a frame in theform of a bezel 7412.

In some embodiments, causing the second portion of the second object tobe presented in the virtual space may include overlaying part of thesecond portion of the second object atop a portion of the frame. Inphysical displays that contain a frame, presentation in the virtualspace may overlap with that frame so as to provide a seamless transitionfrom presentation on the screen of the physical display to the virtualspace. By way of example, as illustrated in FIG. 74, the second portion7454 of the icon 7450 overlaps with the bezel 7412.

Some disclosed embodiments may involve analyzing the image sensorsignals to determine a visual parameter of the first portion of thesecond object as presented on the physical display. Analyzing imagesensor signals to determine a visual parameter may refer to applying asoftware, equation, or algorithm to identify some sort of indiciarelating to characteristics of a captured image. Such indicia mayinclude brightness, contrast, color range, hue, saturation, chromaticadaptation, other color appearance phenomena, or other characteristicsassociated with the captured image. In the above example where a digitalcamera may photograph a physical display, software may be utilized todetermine a brightness level of a portion of a widget disposed on thephysical display. In one example, the image sensor signals or the imagemay be analyzed to detect a region in the image corresponding to thefirst portion of the second object presented on the physical display,for example using a template matching algorithm. Values of the pixels inthe detected region may be analyzed, for example using a statisticalfunction or a histogram, to determine the visual parameter.

In some embodiments, causing the second portion of the second object tobe presented in the virtual space includes setting a display parameterof the second portion of the second object based on the determinedvisual parameter. In one example, the image of a physical display may beanalyzed to determine a particular brightness level of a widget on thephysical display, the brightness of the remaining part of the widget tobe displayed in the virtual space may be set to match the brightnesslevel of the portion of the widget on the physical display. Accordingly,features such as brightness or color scheme of the two portions of thewidget may be matched to, for example, present a seemingly seamlesspresentation of the widget. By way of example, as illustrated in FIG.74, the first portion 7452 and second portion 7454 may be displayed tohave the same or similar, for example, brightness.

Some disclosed embodiments may involve performing further operationsincluding determining that a user of the wearable extended realityappliance is walking away (or otherwise moving away) from the physicaldisplay. A determination that a user is walking (or otherwise moving)away may involve utilizing sensors associated with the wearable extendedreality device to sense that a distance, angle, or orientation betweenthe user and the physical display may be changing. In another example,positioning data or motion data of the wearable extended reality devicemay be analyzed (for example based on a known position of the physicaldisplay) to determine that the user is walking (or otherwise moving)away from the physical display.

In response to the determination that the user of the wearable extendedreality appliance is walking (or otherwise moving) away from thephysical display, some embodiments may involve presenting both the firstportion and the second portion of the second object in the virtual spacein a manner moving with the user while the first object remains on thephysical display. Accordingly, full view of both objects may be observedcompletely by the user regardless of the user's movement, positioning orline of sight to the physical display. In one example, the second objectmay be moved to remain in the same location of a user's field of view asthe user moves and the first object remains in the same location on thephysical display regardless of the movement of the user. For example, asa user steps away from a physical display, the physical display mayappear smaller in a user's field of view. Since (i) the second objectwas initially partially viewable on the physical display and partiallyviewable on the virtual space and (ii) the boundaries of the physicaldisplay subsequently appear to become smaller, the second portion of thesecond object may remain in the same location with regards to the user'sfield of view and the first portion of the second object may also appearnext to the second portion on the virtual space due to the boundaries ofthe physical display appearing to fall away.

Disclosed embodiments may further involve receiving fourth signalsrepresentative of a fourth object initially presented on a firstphysical display, later virtually presented in extended reality, andsubsequently presented on a second physical display; and in response toreceipt of the fourth signals, causing the fourth object to be presentedon the second physical display. The fourth signals, like the first,second, and third signals, may be digital signals or analog signals. Inone example, the fourth signals may represent, for example, sensor data,textual data, voice data, video data, graphical data, geometric data, orany other form of data that provides perceptible information associatedwith the fourth object. In another example, the fourth signals mayinclude an indication that the fourth object is initially presented on afirst physical display, later virtually presented in extended reality,and subsequently presented on a second physical display. In someexamples, receiving the fourth signals may include at least one ofreading the fourth signals from memory, receiving the fourth signalsfrom an external device, receiving the fourth signals from a software orhardware component controlling at least part of the presentation on thephysical display and/or on the second physical display, or receiving thefourth signals from a software or hardware component controlling anextended reality environment including the virtual space. In someembodiments, the fourth object, like the first, second, and thirdobjects, may be an incidence of virtual content. In some examples, thefourth object may be or include an element of a user interface, such asa window, an input control element, a navigation component, aninformation component, an icon, a widget, and so forth. In someexamples, the fourth object may be or include at least one of text,image, video or graphical element (such as a two-dimensional graphicalelement, a two-dimensional projection of a three-dimensional graphicalelement, and so forth).

By way of example, as illustrated in FIGS. 75A-75D, a fourth object 7540may be moved from the first physical display 7510 to the second physicaldisplay 7530 while being visible virtually in virtual space 7520 betweenthis origin and destination. In FIG. 75A, the fourth object 7540 ispresented completely on the first physical display 7510. In FIG. 75B,the fourth object 7540 is presented partially on the first physicaldisplay 7510 and partially virtually in virtual space 7520. In FIG. 75C,the fourth object 7540 is presented completely virtually in virtualspace 7520. In FIG. 75D, the fourth object 7540 is presented completelyon the second physical display 7530.

In some embodiments, causing the fourth object to be presented on thesecond physical display may include transmitting data reflective of thefourth object to a computing device associated with the second physicaldisplay. A computing device may refer to a device having at least oneprocessor configured to execute computer programs, applications,methods, processes, or other software. Since different processingdevices may control the first and second physical displays, dataassociated with the fourth object may be sent to the processing deviceassociated with the second physical display so that the fourth objectmay be presented on the second physical display. In one example, thedata may be transmitted from the device controlling the first physicaldisplay. In another example, the data may be transmitted from acentralized system controlling the extended reality environment, fromthe wearable extended reality appliance, or from another computerizeddevice. In one example, the data may include at least one of an image ora video of the fourth object, a model (two-dimensional orthree-dimensional) of the fourth object, a software controlling thefourth object, parameters for a software controlling the fourth object,or an indication of the fourth object.

Some disclosed embodiments may also involve receiving input signalsindicative of typed text. Input signals indicative of typed text mayrefer to digital or analog signals representing character encoding suchas those in compliance with the American Standard Code for InformationInterchange (ASCII) or with another format. Such input signals may begenerated or sent from a keyboard, a touchpad, or any other devicecapable of selecting characters.

Some disclosed embodiments may additionally involve concurrentlydisplaying the typed text on a first display and on a second display,wherein the second display is an extended reality display area locatedadjacent to a keyboard. Concurrently displaying the typed text may referto presenting the typed text simultaneously on two or more displays,duplicating the text in to each one of the two of more displays. Asdiscussed above the term keyboard may refer to any device capable ofselecting characters of text, such as a physical keyboard, a virtualkeyboard, etc. In some embodiments, the first display may be thephysical display. For example, the typed text may be displayed in a textediting application, in a text input element, or in any other elementpresented on the physical display. In some embodiments, the firstdisplay may be a virtual display other than the second display. Forexample, the typed text may be displayed in a text editing application,in a text input element, or in any other element presented on thevirtual display.

Some disclosed embodiments may involve receiving fourth signalsrepresentative of a fourth object having a first portion and a secondportion, initially the fourth object is wholly presented on the physicaldisplay. The fourth signals, like the first, second, and third signals,may be digital signals or analog signals. In one example, the fourthsignals may represent, for example, sensor data, textual data, voicedata, video data, graphical data, geometric data, or any other form ofdata that provides perceptible information. In another example, thefourth signals may include an indication that the fourth object have afirst portion and a second portion, and/or that the fourth object isinitially wholly presented on the physical display. In yet anotherexample, the fourth signals may include information associated with thefirst portion and/or the second portion of the fourth object. In someexamples, receiving the fourth signals may include at least one ofreading the fourth signals from memory, receiving the fourth signalsfrom an external device, receiving the fourth signals from a software orhardware component controlling at least part of the presentation on thephysical display, or receiving the fourth signals from a software orhardware component controlling an extended reality environment includingthe virtual space. In some embodiments, the fourth object, like thefirst, second, and third objects, may be an incidence of virtualcontent. In some examples, the fourth object may be or include anelement of a user interface, such as a window, an input control element,a navigation component, an information component, an icon, a widget, andso forth. In some examples, the fourth object may be or include at leastone of text, image, video or graphical element (such as atwo-dimensional graphical element, a two-dimensional projection of athree-dimensional graphical element, and so forth). This virtual contentmay, like the previously described objects, may be viewable in itsentirety by the user. By way of example, as illustrated in FIG. 75A, thefourth object 7540 has a first portion 7542 and a second portion 7544,which are both initially presented completely on the physical display7510.

Some disclosed embodiments may involve receiving fifth signalsindicating that the fourth object is moved to a position in which thefirst portion of the fourth object is presented on the physical displayand the second portion of the fourth object is extending beyond theborder of the physical display. The fifth signals, like the first,second, third, and fourth signals, may be digital signals or analogsignals. In one example, the fifth signals may include an indicationthat the fourth object have a first portion and a second portion, and/orthat the fourth object is moved to the position in which the firstportion of the fourth object is presented on the physical display andthe second portion of the fourth object is extending beyond the borderof the physical display. In yet another example, the fifth signals mayinclude information associated with the first portion and/or the secondportion of the fourth object. In some examples, receiving the fifthsignals may include at least one of reading the fifth signals frommemory, receiving the fifth signals from an external device, receivingthe fifth signals from a software or hardware component controlling atleast part of the presentation on the physical display, or receiving thefifth signals from a software or hardware component controlling anextended reality environment including the virtual space. By way ofexample, as illustrated in FIG. 75B, the first portion 7542 may remaindisplayed on the physical display 7510 and a second portion 7544 mayextend beyond the border of the physical display 7510.

In response to the fifth signals, some embodiments may involve causingthe second portion of the fourth object to be presented via the wearableextended reality appliance in the virtual space while the first portionof the fourth object is presented on the physical display. By way ofexample, as illustrated in FIG. 75B, the first portion 7542 may remaindisplayed on the physical display 7510 and a second portion 7544 may bedisplayed in the virtual space 7520.

Some disclosed embodiments may involve receiving sixth signalsindicating that the fourth object is wholly moved beyond the border ofthe physical display. The sixth signals, like the first, second, third,fourth, and fifth signals, may be digital signals or analog signals. Inone example, the sixth signals may include an indication that the fourthobject is wholly moved beyond the border of the physical display. Insome examples, receiving the sixth signals may include at least one ofreading the sixth signals from memory, receiving the sixth signals froman external device, receiving the sixth signals from a software orhardware component controlling at least part of the presentation on thephysical display, or receiving the sixth signals from a software orhardware component controlling an extended reality environment includingthe virtual space. The fourth object may, like the previously describedobjects, be moved from one location to another and that other locationmay be beyond the display boundaries of the display the content wasinitially presented on. By way of example, as illustrated in FIG. 75C,the fourth object 7540 has moved completely beyond the border of thephysical display 7510.

In response to receipt of the sixth signals, some embodiments mayinvolve causing the fourth object to be wholly presented via thewearable extended reality appliance in the virtual space. This fourthobject may, like the previously described objects, may be viewable inits entirety by the user even after moving beyond the display boundariesof the display the content was initially presented on. By way ofexample, as illustrated in FIG. 75C, the fourth object 7540 has movedcompletely into the virtual space 7520.

Other disclosed embodiments may include a system for augmenting aphysical display with an extended reality display. the system mayinclude at least one processor configured to: receive first signalsrepresentative of a first object fully presented on a physical display;receive second signals representative of a second object having a firstportion presented on the physical display and a second portion extendingbeyond a border of the physical display; receive third signalsrepresentative of a third object initially presented on the physicaldisplay and subsequently wholly moved beyond the border of the physicaldisplay; in response to receipt of the second signals, cause the secondportion of the second object to be presented via a wearable extendedreality appliance in a virtual space while the first portion of thesecond object is presented on the physical display; and in response toreceipt of the third signals, cause the third object to be whollypresented via the wearable extended reality appliance in the virtualspace following the third object having been wholly presented on thephysical display.

In some examples, the extent of the virtual space may be selected basedon a presence of other physical objects in the vicinity of the physicaldisplay. For example, the physical display may be adjacent to anotherphysical display, to a wall, or any other physical objects. In oneembodiment, image data captured using an image sensor (for example, animage sensor included in the wearable extended reality appliance, animage sensor external to the wearable extended reality appliance, etc.)may be analyzed to detect a physical object in the vicinity of thephysical display. The image data may be analyzed using at least one ofan object detection algorithm, an object recognition algorithm, asemantic segmentation algorithm, and any other relevant algorithm.

In another embodiment, radar, Lidar or sonar sensor may be used todetect a physical object in the vicinity of the physical display. Forexample, the virtual space may be selected so that it does not overlapwith the detected physical object. The virtual space may be selected sothat it does not hide at least part of the detected physical object. Inyet another example, the virtual space may be selected so that it is notat least partly hidden by the detected physical object. In someexamples, in response to a first type of physical object (such as asecond physical display, a lamp, etc.), the virtual space may beselected so that it does not overlap with the detected physical object,and in response to a second type of the physical object (such as a wall,a vase, etc.), the virtual space may be selected so that it overlapswith the detected physical object. In one implementation, the type ofthe physical object may be determined using an object recognitionalgorithm.

Other disclosed embodiments may include a method for augmenting aphysical display with an extended reality display. By way of example,FIG. 76 shows a flowchart illustrating an exemplary method 7600 foraltering a perspective of a scene, consistent with some embodiments ofthe present disclosure. Method 7600 may include a step 7610 of receivingfirst signals representative of a first object fully presented on aphysical display. Method 7600 may include a step 7612 of receivingsecond signals representative of a second object having a first portionpresented on the physical display and a second portion extending beyonda border of the physical display. Method 7600 may include a step 7614of, in response to receipt of the second signals, causing the secondportion of the second object to be presented via a wearable extendedreality appliance in a virtual space while the first portion of thesecond object is presented on the physical display. Method 7600 mayinclude a step 7616 of, in response to receipt of the third signals,causing the third object to be wholly presented via the wearableextended reality appliance in the virtual space following the thirdobject having been wholly presented on the physical display.

Unless otherwise defined, all technical and scientific terms used hereinhave the same meaning as commonly understood by one of ordinary skill inthe art. The materials, methods, and examples provided herein areillustrative only and not intended to be limiting.

Implementation of the method and system of the present disclosure mayinvolve performing or completing certain selected tasks or stepsmanually, automatically, or a combination thereof. Moreover, accordingto actual instrumentation and equipment of preferred embodiments of themethod and system of the present disclosure, several selected steps maybe implemented by hardware (HW) or by software (SW) on any operatingsystem of any firmware, or by a combination thereof. For example, ashardware, selected steps of the disclosure could be implemented as achip or a circuit. As software or algorithm, selected steps of thedisclosure could be implemented as a plurality of software instructionsbeing executed by a computer using any suitable operating system. In anycase, selected steps of the method and system of the disclosure could bedescribed as being performed by a data processor, such as a computingdevice for executing a plurality of instructions.

Various implementations of the systems and techniques described here canbe realized in digital electronic circuitry, integrated circuitry,specially designed ASICs (application specific integrated circuits),computer hardware, firmware, software, and/or combinations thereof.These various implementations can include implementation in one or morecomputer programs that are executable and/or interpretable on aprogrammable system including at least one programmable processor, whichmay be special or general purpose, coupled to receive data andinstructions from, and to transmit data and instructions to, a storagesystem, at least one input device, and at least one output device.

The systems and techniques described here can be implemented in acomputing system that includes a back end component (e.g., as a dataserver), or that includes a middleware component (e.g., an applicationserver), or that includes a front end component (e.g., a client computerhaving a graphical user interface or a Web browser through which a usercan interact with an implementation of the systems and techniquesdescribed here), or any combination of such back end, middleware, orfront end components. The components of the system can be interconnectedby any form or medium of digital data communication (e.g., acommunication network). Examples of communication networks include alocal area network (“LAN”), a wide area network (“WAN”), and theInternet. The computing system can include clients and servers. A clientand server are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

While certain features of the described implementations have beenillustrated as described herein, many modifications, substitutions,changes and equivalents will now occur to those skilled in the art. Itis, therefore, to be understood that the appended claims are intended tocover all such modifications and changes as fall within the scope of theimplementations. It should be understood that they have been presentedby way of example only, not limitation, and various changes in form anddetails may be made. Any portion of the apparatus and/or methodsdescribed herein may be combined in any combination, except mutuallyexclusive combinations. The implementations described herein can includevarious combinations and/or sub-combinations of the functions,components and/or features of the different implementations described.

The foregoing description has been presented for purposes ofillustration. It is not exhaustive and is not limited to the preciseforms or embodiments disclosed. Modifications and adaptations of theembodiments will be apparent from consideration of the specification andpractice of the disclosed embodiments. For example, the describedimplementations include hardware and software, but systems and methodsconsistent with the present disclosure may be implemented as hardwarealone.

It is appreciated that the above-described embodiments can beimplemented by hardware, or software (program codes), or a combinationof hardware and software. If implemented by software, it can be storedin the above-described computer-readable media. The software, whenexecuted by the processor can perform the disclosed methods. Thecomputing units and other functional units described in the presentdisclosure can be implemented by hardware, or software, or a combinationof hardware and software. One of ordinary skill in the art will alsounderstand that multiple ones of the above-described modules/units canbe combined as one module or unit, and each of the above-describedmodules/units can be further divided into a plurality of sub-modules orsub-units.

The block diagrams in the figures illustrate the architecture,functionality, and operation of possible implementations of systems,methods, and computer hardware or software products according to variousexample embodiments of the present disclosure. In this regard, eachblock in a flowchart or block diagram may represent a module, segment,or portion of code, which includes one or more executable instructionsfor implementing the specified logical functions. It should beunderstood that in some alternative implementations, functions indicatedin a block may occur out of order noted in the figures. For example, twoblocks shown in succession may be executed or implemented substantiallyconcurrently, or two blocks may sometimes be executed in reverse order,depending upon the functionality involved. Some blocks may also beomitted. It should also be understood that each block of the blockdiagrams, and combination of the blocks, may be implemented by specialpurpose hardware-based systems that perform the specified functions oracts, or by combinations of special purpose hardware and computerinstructions.

In the foregoing specification, embodiments have been described withreference to numerous specific details that can vary from implementationto implementation. Certain adaptations and modifications of thedescribed embodiments can be made. Other embodiments can be apparent tothose skilled in the art from consideration of the specification andpractice of the invention disclosed herein. It is intended that thespecification and examples be considered as example only, with a truescope and spirit of the invention being indicated by the followingclaims. It is also intended that the sequence of steps shown in figuresare only for illustrative purposes and are not intended to be limited toany particular sequence of steps. As such, those skilled in the art canappreciate that these steps can be performed in a different order whileimplementing the same method.

It will be appreciated that the embodiments of the present disclosureare not limited to the exact construction that has been described aboveand illustrated in the accompanying drawings, and that variousmodifications and changes may be made without departing from the scopethereof. And other embodiments will be apparent to those skilled in theart from consideration of the specification and practice of thedisclosed embodiments disclosed herein. It is intended that thespecification and examples be considered as exemplary only, with a truescope and spirit of the disclosed embodiments being indicated by thefollowing claims.

Moreover, while illustrative embodiments have been described herein, thescope includes any and all embodiments having equivalent elements,modifications, omissions, combinations (e.g., of aspects across variousembodiments), adaptations or alterations based on the presentdisclosure. The elements in the claims are to be interpreted broadlybased on the language employed in the claims and not limited to examplesdescribed in the present specification or during the prosecution of theapplication. These examples are to be construed as non-exclusive.Further, the steps of the disclosed methods can be modified in anymanner, including by reordering steps or inserting or deleting steps. Itis intended, therefore, that the specification and examples beconsidered as exemplary only, with a true scope and spirit beingindicated by the following claims and their full scope of equivalents.

1-240. (canceled)
 241. A non-transitory computer readable mediumcontaining instructions for performing operations configured to augmenta physical display with an extended reality display, the operationscomprising: receiving first signals representative of a first objectfully presented on a physical display; receiving second signalsrepresentative of a second object having a first portion presented onthe physical display and a second portion extending beyond a border ofthe physical display; receiving third signals representative of a thirdobject initially presented on the physical display and subsequentlywholly moved beyond the border of the physical display; in response toreceipt of the second signals, causing the second portion of the secondobject to be presented via a wearable extended reality appliance in avirtual space while the first portion of the second object is presentedon the physical display; and in response to receipt of the thirdsignals, causing the third object to be wholly presented via thewearable extended reality appliance in the virtual space following thethird object having been wholly presented on the physical display. 242.The non-transitory computer readable medium of claim 241, wherein atleast one of the first object, the second object, or the third objectincludes at least one of a widget or an icon of an application.
 243. Thenon-transitory computer readable medium of claim 241, wherein the secondobject partially overlaps with at least one of the first object or thethird object.
 244. The non-transitory computer readable medium of claim241, wherein the first object, the second object, and the third objectare concurrently presented on the physical display and in the virtualspace.
 245. The non-transitory computer readable medium of claim 241,wherein the physical display is part of an input device configured togenerate text to be presented on the physical display.
 246. Thenon-transitory computer readable medium of claim 241, wherein thephysical display is part of an input device configured to generate textto be presented in the virtual space.
 247. The non-transitory computerreadable medium of claim 241, wherein at least one of the first signals,the second signals, or the third signals are received from an operatingsystem controlling the physical display.
 248. The non-transitorycomputer readable medium of claim 241, wherein at least one of the firstsignals, the second signals, or the third signals are received from apointing device associated with the wearable extended reality appliance.249. The non-transitory computer readable medium of claim 241, whereinthe operations further include: receiving image sensor signalsrepresenting an image of the physical display; ascertaining a boundingedge of the physical display; and registering the virtual space with thephysical display based on the ascertained bounding edge.
 250. Thenon-transitory computer readable medium of claim ______, wherein thephysical display includes a frame defining the bounding edge, andwherein causing the second portion of the second object to be presentedin the virtual space includes overlaying part of the second portion ofthe second object atop a portion of the frame.
 251. The non-transitorycomputer readable medium of claim ______, wherein the operations furtherinclude analyzing the image sensor signals to determine a visualparameter of the first portion of the second object as presented on thephysical display, and wherein causing the second portion of the secondobject to be presented in the virtual space includes setting a displayparameter of the second portion of the second object based on thedetermined visual parameter.
 252. The non-transitory computer readablemedium of claim 241, wherein the operations further include: determiningthat a user of the wearable extended reality appliance is walking awayfrom the physical display; and in response to the determination, causingboth the first portion and the second portion of the second object to bepresented in the virtual space and to move with the user while the firstobject remains on the physical display.
 253. The non-transitory computerreadable medium of claim 241, wherein the operations further include:receiving fourth signals representative of a fourth object initiallypresented on a first physical display, later virtually presented inextended reality, and subsequently presented on a second physicaldisplay; and in response to receipt of the fourth signals, causing thefourth object to be presented on the second physical display.
 254. Thenon-transitory computer readable medium of claim ______, wherein causingthe fourth object to be presented on the second physical displayincludes transmitting data reflective of the fourth object to acomputing device associated with the second physical display.
 255. Thenon-transitory computer readable medium of claim 241, wherein theoperations further include: receiving input signals indicative of typedtext; and concurrently displaying the typed text on a first display andon a second display, wherein the second display is an extended realitydisplay area located adjacent to a keyboard.
 256. The non-transitorycomputer readable medium of claim ______, wherein the first display isthe physical display.
 257. The non-transitory computer readable mediumof claim ______, wherein the first display is a virtual display otherthan the second display.
 258. The non-transitory computer readablemedium of claim 241, wherein the operations further include: receivingfourth signals representative of a fourth object having a first portionand a second portion, initially the fourth object is wholly presented onthe physical display; receiving fifth signals indicating that the fourthobject is moved to a position in which the first portion of the fourthobject is presented on the physical display and the second portion ofthe fourth object is extending beyond the border of the physicaldisplay; in response to the fifth signals, causing the second portion ofthe fourth object to be presented via the wearable extended realityappliance in the virtual space while the first portion of the fourthobject is presented on the physical display; receiving sixth signalsindicating that the fourth object is wholly moved beyond the border ofthe physical display; and in response to receipt of the sixth signals,causing the fourth object to be wholly presented via the wearableextended reality appliance in the virtual space.
 259. A system foraugmenting a physical display with an extended reality display, thesystem comprising: at least one processor configured to: receive firstsignals representative of a first object fully presented on a physicaldisplay; receive second signals representative of a second object havinga first portion presented on the physical display and a second portionextending beyond a border of the physical display; receive third signalsrepresentative of a third object initially presented on the physicaldisplay and subsequently wholly moved beyond the border of the physicaldisplay; in response to receipt of the second signals, cause the secondportion of the second object to be presented via a wearable extendedreality appliance in a virtual space while the first portion of thesecond object is presented on the physical display; and in response toreceipt of the third signals, cause the third object to be whollypresented via the wearable extended reality appliance in the virtualspace following the third object having been wholly presented on thephysical display.
 260. A method for augmenting a physical display withan extended reality display, the method comprising: receiving firstsignals representative of a first object fully presented on a physicaldisplay; receiving second signals representative of a second objecthaving a first portion presented on the physical display and a secondportion extending beyond a border of the physical display; receivingthird signals representative of a third object initially presented onthe physical display and subsequently wholly moved beyond the border ofthe physical display; in response to receipt of the second signals,causing the second portion of the second object to be presented via awearable extended reality appliance in a virtual space while the firstportion of the second object is presented on the physical display; andin response to receipt of the third signals, causing the third object tobe wholly presented via the wearable extended reality appliance in thevirtual space following the third object having been wholly presented onthe physical display.